This is the first in a short series where we share results from controlled experiments we have been running on how AI answer engines (Claude, ChatGPT, and similar tools) decide what to say. The full methodology and data can be found here: When Does the Model Look It Up?
AI platforms like ChatGPT and Claude commonly rely on both the data used to train the LLMs that power them, as well as search engines which they use as tools to retrieve live information from the web. If you ask one of these tools about a large national brand, it can usually answer from memory, because it saw that brand many times during training. If you ask about an independent brokerage in a mid-size city, it has no such memory. Most real estate professionals fall into the second case, so the question we set out to answer is a practical one: when the model has no prior knowledge of a business and it goes out to search, what actually determines the answer it gives?
How we tested it
In order to answer this cleanly, we needed to control the one thing you can never control when you simply watch a live chatbot, which is exactly what the model sees when it looks something up. So we gave the model a search tool, but every result it received came from pages we wrote ourselves.
We then asked it about three kinds of business: ones it clearly knows (for example Redfin and Zillow), real but obscure local firms we found around the country, and a set of businesses we invented. Before running anything, we checked which was which by asking the model, with no search allowed, what it knew about each. The national brands produced specific facts every time. The obscure and invented ones produced "I'm not familiar with them" almost every time. So we knew, going in, exactly what was and was not already in the model's memory.
The two results that matter
The first result is what happens with no ability to search. Asked about a business it does not know, the model does the sensible thing and declines. In our runs the invented businesses drew a refusal essentially all of the time, and the real but obscure ones drew a refusal in the large majority of cases. That is the entire starting position for most local businesses today - not a bad answer, but no answer at all.
The second result is what happens the moment the model can search our pages. The refusals almost entirely disappear, and the model goes on to state 92 to 93 percent of whatever facts we had written on those pages - founding years, team sizes, owners' names, service details - confidently, and with no hedging. In other words, for a business the model has no prior knowledge of, the page it retrieves does not just inform the answer. It becomes the answer. There is no competing memory for that page to argue with, so whatever the model finds is what it repeats.
Why this matters more than any ranking tactic
Most discussion of "AI optimization" borrows the anxieties of SEO - keywords, rankings, and gaming an algorithm. For a business the model already knows, some of that nuance does apply, and we will cover it in a later article. But for the businesses that make up most of the market, the ones no model has memorized, the situation is both simpler and more urgent than any ranking tactic. If the model can find a clear, specific, factual page about you, it will describe you with confidence. If it can find nothing, it will say you do not exist.
There is a flip side worth stating plainly. Because the model has no independent knowledge of an unfamiliar business, it also has no way to check what it finds. In our experiment it repeated our planted facts whether or not they were accurate. For an unfamiliar business, the most authoritative page the model can reach is, as far as the model is concerned, the truth. That is a responsibility as much as an opportunity, because the pages that represent you - your own site, your profiles, the third-party listings that mention you - are effectively the model's memory of you.
What we suggest
The takeaway here is not a trick. It is a change in what you optimize for.
- Be findable first. Before anything else, make sure clear, factual pages about your business exist and can be retrieved. An unknown business with no reachable page is simply absent from AI answers.
- Be specific. Vague pages give the model nothing to work with. Founding year, who runs the firm, size, service area, and what you actually do are the kinds of details that get repeated.
- Keep your own facts accurate everywhere. Since the model repeats what it finds, the accuracy of your reachable pages is the accuracy of what the AI tells people about you.
We are publishing the full methodology and data behind these experiments, because this space is full of confident claims and short on controlled measurement. This is the first result. The next one looks at the businesses the model does know, and shows that even a household name's facts can be overwritten by a single page the model happens to find.
If you would like help thinking through how your own pages read to these tools, let us know and we are happy to work through it with you.