This is the second of three posts on our portal-fidelity study. The first post measured how well a script calling OpenAI's API can stand in for chatgpt.com. This one does the same for Claude, where the answer turned out to be different in kind. The full study is here: How Close Can an API Get to the Chat Window?
For ChatGPT, the gap between a script and the chat window is a sampling gap. The script attaches the vendor's search tool, gets roughly the same kind of pages the product gets, and lands on a different draw from the same pile. We expected Claude to behave the same way, because the ingredients looked the same: Anthropic exposes the model the free claude.ai plan uses (Sonnet 5.5) through its API, publishes the system prompt the product runs under, and offers a hosted web search tool.
Same query, different web
We asked a hundred local questions ("best realtor in Reno, Nevada" and the like) on claude.ai, signed in to a free account in a private, memory-free chat, and through the API in a ladder of configurations that added the published system prompt, a location, the product's "medium" effort level, and a page-fetch tool. The businesses the API named overlapped with the businesses the product named by about one in six, measured as the share of shared names across both lists. That is close to the overlap we get between claude.ai and ChatGPT, and significantly below the roughly one in two that two asks of claude.ai itself share. Configuration did not change this: not the prompt, not the location, not the operator's actual city, not three other models. There may be a silver lining here: if you ask enough times, you may get close to the population of businesses the engine draws on.
Because both surfaces show their work, we could see where the split happens. The model wrote nearly the same search query on both surfaces, as you would expect from the same model reading the same question. What came back was different. The pages the API's search tool returned overlapped with the pages the product's search tool returned by about one in seven, while two API runs of the same question returned almost identical pages (nine in ten). So the API's tool is stable (nine in ten), the product's tool is just as stable (nine in ten), but they are stable about different things.
The two tools also have different tastes. Across fifty questions the API's tool returned a Yelp page in every single answer and a U.S. News agent page in four of ten. The product's tool returned none of these. It leaned instead on Expertise.com, the referral directory Clever, RealTrends rankings and local ranking sites, and for the "best realtor" questions on press-release wires, which the API never surfaced. The answers follow the pages, so the API cites Yelp and U.S. News, the product cites Clever, RealTrends and the wires, and they name different people. The two surfaces agree best where their sources happen to coincide, as for mortgage lenders, and worst on the plain "best realtor" question.
There is one more layer the product has and the API does not. For more than half of the home inspector questions claude.ai did not search the web at all. It used a places tool that returns Google business records, put the businesses on a map, and wrote its answer from their ratings and reviews, phone numbers included. ChatGPT has the equivalent, the map card that appears under many local answers. No API reconstruction of either product can see that layer.
This is not a memory effect
The first dozen answers we captured were in ordinary chats on an account with Anthropic's memory feature switched on, and that account has a long history of conversations about real estate. We worried that memory had coloured the answers. So we re-asked the same questions in private chats, where memory is off, and compared. The memory-on answers agreed with the private ones about as well as repeated asks of either of them did. Thus, personalization due to an existing chat history seemed to have little effect on the ranking of businesses.
What this means if you track your visibility on Claude
- Be careful to ask which surface a report measured, the claude.ai product or the API, when you look at reports that may claim Claude recommends you.
- The sources that matter on claude.ai are, at least for local professionals right now, the ranking and referral directories (Expertise, Clever, RealTrends and their local equivalents), with press-release wires showing up for the plain "best realtor" question, and Google business ratings where the places tool fires. An agent with those profiles is in the product's candidate set; one with only a strong Yelp presence is in the API's.
- The best way to measure the product is to ask the product.