At Orchid Box, we investigate how AI assistants find their information. Funded by our own resources, we ran an exploratory research experiment using AI agents to build a network of 12 experimental sites. We then measured crawler requests, search activity and citations in sampled Gemini answers. Our aim was to understand how crawlers discover new websites and whether articles from those websites subsequently appear in AI-generated answers. We watched which AI bots came to read them and whether any assistant cited them.
Every major AI lab sends more than one bot to your site. Each bot has a different job. They fall into three groups. Automated services may request website pages for different purposes.
Broadly, these include activity associated with model development, search or discovery, and fetching a page in response to a user request. These categories do not describe every operator’s systems, and the name presented by a requesting bot does not by itself establish who operated it or how the retrieved material was subsequently used.
| Bot type | What it does | Main bots by lab |
|---|---|---|
| Training bots | Collect pages to train future models. What they take may shape a model’s answers months later. It does not lead to a link today. | OpenAI: GPTBot Anthropic: ClaudeBot Meta: Meta-ExternalAgent |
| Search bots | Build the index an assistant search when it answers a question. This index decides which pages an assistant can cite. | OpenAI: OAI-SearchBot Anthropic: Claude-SearchBot Perplexity: PerplexityBot |
| User fetch bots | Visit a page when an assistant is prompted about it. A request from one of these is still a bot fetching the page, not a person reading it. | OpenAI: ChatGPT-User Anthropic: Claude-User Perplexity: Perplexity-User Meta: Meta-ExternalFetcher Google: Google-Agent |
These examples and classifications should be read subject to the operators’ current documentation. User-agent names can be imitated and do not authenticate the operator.
Bots now make up more than half of all web traffic. In early June 2026 Cloudflare put the figure at 57.5% of webpage requests. Among crawlers, Googlebot alone made about 27% of requests in May 2026. GPTBot made about 11% and ClaudeBot about 10%. Automated requests account for a substantial part of activity reported by some web infrastructure providers. Those figures describe the providers’ own datasets and definitions, not necessarily traffic across the entire web or activity at our experimental sites.
Few of those visits send anyone back. In July 2025 Cloudflare measured how many pages each lab crawled for every visitor it referred to. Crawling and referred visits measure different things. A service may request many pages without sending an identifiable visitor back to the publisher. External crawl-to-referral comparisons provide context, but they do not establish what happened at our sites.
| Operator | Pages crawled per referred visitor, July 2025 |
|---|---|
| Anthropic | About 38,000 |
| OpenAI | About 1,091 |
| About 5 |
Your analytics will not show you any of this.
Google Analytics and similar tools count people. They run in the browser as a script and bots do not run it. So, your analytics dashboard will never show you an AI crawler. Browser-based analytics can miss automated requests because some crawlers do not execute the scripts those tools rely on. An analytics dashboard therefore does not necessarily capture all crawler activity.
The only place bots show up is your server logs. That is first-party data, and you need it for a test like this. Without logs you can only guess.
Before AI agents, a test like this was slow and costly. You had to set up several sites and fill them with content. Then you had to keep them running for weeks and study how bots crawled them. You needed to track how often each bot came and which pages it wanted. You also needed to track how long a fact took to show up in an AI answer.
Claude Fable 5 was the right tool for the job. We used it almost as an editorial black box. We gave it a goal and it built the sites, set up the editorial desk and kept content flowing. It ran a team of agents that could research, write and publish at a scale and speed a small human team could not match.
Setting up several sites, preparing content, maintaining them and monitoring requests would normally take substantial practical effort. The agents handled most of it. A separate checking stage compared each draft with its source brief.
This was an exploratory experiment, not a controlled study capable of establishing how every AI service selects its sources.
Claude Fable 5 suggested the best way to fill the sites was an editorial desk run by several agents. AI agents assisted with an experimental editorial workflow that produced three types of content from publisher feeds and extracts, institutional datasets, and public tender notices:
| Story type | Sources | How it was made |
|---|---|---|
| Reported news | Publisher feeds | Several feeds often carry the same event, so the desk grouped them. It fetched up to 4,000 characters of paragraph text per story from each publisher’s page. A writer agent worked from a brief with up to four source extracts of 1,200 characters each. A fact checker agent then compared the draft with the brief. |
| Data stories | Eurostat, ECB, SMARD, TED | Trend series came with figures attached. A measured figure counted as evidence on its own, so these ran as the site’s own analysis. |
| Register stories | TED, Find a Tender | One public tender notice per story. These fed the procurement beats. |
The editorial desk graded every source it used. Primary sources were institutions, regulators, and statistics offices. Secondary sources were a named list of established news organisations. Other publishers were graded “unknown” but still accepted. Aggregators such as Google News, Yahoo and MSN were never accepted as a source by the desk. Some sites also pulled content directly from publishers outside the desk, so this grading did not apply to everything published on the network.
The workflow distinguished institutional sources, published reporting and aggregators. That classification described the source type; it did not establish permission to copy or republish the material. Public accessibility and AI-generated wording do not, by themselves, resolve questions about copyright, attribution, or other rights.
Here is how the main labs behaved in the server logs available for the 12 sites in the study. The following table summarises requests recorded under user-agent names associated with the listed services. It does not authenticate the operators or establish subsequent indexing, training or use in an answer.
| Lab | Requests | Main focus | What stood out |
|---|---|---|---|
| OpenAI | 65,551 | Search | OAI-SearchBot made 64% of requests and got a real page 78% of the time. ChatGPT-User made 5,712 requests. |
| Anthropic | 49,437 | Training | ClaudeBot made 99% of requests and read robots.txt 1,593 times. Claude-SearchBot got 68 pages from 165 requests and Claude-User 128 from 489. |
| Meta | About 15,400 | Training | Took about 3,400 pages. Never read robots.txt. |
Totals, classifications, and observation periods remain subject to confirmation against the underlying logs. Requests are not equivalent to unique visitors or successful retrievals.
Here are our findings from our research. The views expressed are entirely based upon that research.
OpenAI cared most about search. About 64% of its requests came from OAI-SearchBot. That is the bot that decides what ChatGPT can cite. It got a real page 78% of the time. ChatGPT-User is the most interesting line. It requested our pages 5,712 times. These were automated requests. We could not link any of them to a real reader. 2,518 of those requests got a “down” (503) page.
Among requests bearing OpenAI-associated names, the largest share used OAI-SearchBot. We also recorded requests bearing GPTBot and ChatGPT-User names. These records do not establish what was indexed, used for training, or cited. We did not rely on referrer data, because referrers can be faked.
Anthropic cared most about training. ClaudeBot, its training crawler, made 99% of Anthropic’s requests. It read robots.txt 1,593 times, about once every 31 requests, more than any other bot. Anthropic’s search and user-fetch bots barely appeared. Claude-SearchBot made 165 requests and got a page 68 times. Claude-User made 489 requests and got a page 128 times. We could not verify the operator of these smaller groups of requests.
We did not measure Google’s crawlers. Gemini has no crawler of its own. It answers from Google’s search index, which is built for Google Search as a whole. So Google’s requests could not be attributed to Gemini.
Meta’s training crawler was busy and quiet. It made about 15,400 requests and took about 3,400 pages. It never once read robots.txt. These records do not establish that the material was used for training. Any statement about robots.txt requests is limited to the records, sites and period examined and does not establish compliance or non-compliance.
But those findings give rise to concerns that some requests used bot names falsely. CCBot, Bytespider, Claude-SearchBot, Claude-User and Perplexity-User failed more often than not, but each received some real pages. They got a page 31 times out of 76, 127 out of 749, 68 out of 165, 128 out of 489 and 36 out of 283. Some requests bore names whose operator identity or purpose we could not establish confidently. Unsuccessful responses, including 403, 404 and 405 responses, do not by themselves prove impersonation. Counting requests by user-agent name alone may therefore misclassify activity. Some of this may be scanners using famous bot names. Counting bots by name alone may overstate real AI crawling.
| Question | What we found |
|---|---|
| How often do AI bots crawl a brand-new site? | Regularly. In the server logs available for the 12 sites, OpenAI and Anthropic made about 115,000 requests between them over the logged period (8 August to 8 October 2026, varying by site). OpenAI came about a third more often than Anthropic. |
| How long does it take for an AI to cite content from a new site? | We never saw a citation of any page. Gemini was checked daily from the start of each campaign (4, 8 or 18 September) to 22 September and never cited one. The only finding is that none came within the test period. |
| Does country matter? | Inconclusive. The data was not statistically significant, so we found no compelling evidence either way. |
| Does the language of the content make a difference? | On the multilingual sites, English pages drew the vast majority of OpenAI and Anthropic visits. This may partly reflect how many pages existed in each language. The split is below. |
Share of OpenAI and Anthropic visits by page language Provisional share of recorded OpenAI-associated and Anthropic-associated requests by page language
| Language | Share of visits |
|---|---|
| English | 93% |
| German | 5% |
| French | 2% |
These figures describe the distribution of recorded requests, not a demonstrated language preference or causal effect. The denominators, classifications and observation period require verification.
We saw no network citations in the Gemini answers sampled. No answer cited a network article.
Across seven monitored campaigns, no sampled Gemini answer cited a network article. This is a finding about those answers, not a claim that no AI service ever cited a network page.
Citations are hard to track, even with tools such as LLM Scout. Assistants answer differently each time. Answers change by country and by user. Most citations never lead to a click. A tool can only sample answers. So, it can never prove a page was never cited. Citation monitoring samples particular answers. Results may vary between prompts, users, locations and times, and a citation may not generate a click. The monitoring therefore cannot establish that a page was never cited elsewhere.
Search traffic was minimal. Google Search Console data for one of the sites showed 20 search clicks across 25 pages between 8 June and 4 October 2026. These are clicks, not unique visitors.
Side note: Ahrefs, a third-party SEO tool, estimated zero organic visits and zero traffic value for all 12 domains. This is an outside estimate, not part of our study data, and it does not prove that no visits took place.
Operators of websites, publishers, and companies whose information was accessed or used as part of this experiment may have questions or concerns. We recognise the time, expertise and resources involved in producing news, data, and other information. The naming of the experimental sites in the network was based on available domains and names accessed via AI assistants. We hope this publication provides a clear explanation and rationale about the experimental nature or status of the sites.
The network failed. We never saw a network page cited in an AI answer. We could not identify any visitor arriving from an AI assistant. Most of the activity we could identify in our logs was automated crawling.
We believe that setting up new sites and filling them with AI-generated content from different sources is not the right way to get cited by AI assistants or to get your brand message into Google. The experiment was designed to investigate discovery and visibility, rather than to generate revenue from the network. We wish to make clear that this experiment generated no revenue and carried no advertisements or paid placements. We accept that those facts help explain its context, but do not resolve questions about sourcing, attribution, or site presentation.
Shortcuts like this clearly do not work. AI assistants and Google trust established sites that have been around for a while. Those sites have built up a history, links, and real readers over time. A new network cannot fake that.
Google has been clear about this. It has run three spam updates in 2026 alone, in March, June and August. Scaled content abuse has been a main target throughout. Sites caught by these updates can lose their rankings, see pages demoted or be dropped from Google’s results altogether. Some also receive a manual action, a penalty applied by Google’s own reviewers.
We have published these findings (a) to explain the rationale behind our experiment (b) to present the findings of that experiment and (c) to respond to some queries we have received about our experiment. We apologise to any organisation offended by being included in this experiment without their knowledge, consent, or attribution. We did not intend to cause any offence or confusion.
