First of three posts on a question that sits underneath every "AI visibility" report, including our own: when a tool checks what ChatGPT or Claude says about a business, is it looking at the same system the customer is? The full study is here: How Close Can an API Get to the Chat Window?
Many tools that monitor AI answers, and a large amount of research on them, use a script to call the vendor's programming interface (the API) in an attempt to replicate answers that chat platforms like ChatGPT and Claude provide. The assumption underneath is that the script and the chat window give the same, or at least a similar, answer. For a real estate agent reading a report that says "ChatGPT recommends you in Boise", that assumption can be very important.
We set out to measure it. In order to do that we first had to measure something simpler, which turned out to be the more useful number.
Ask the same question three times
We wrote 100 questions of the kind a consumer asks: "best realtor in Boise, Idaho", "best home inspector in Reno, Nevada", "best mortgage lender in Savannah, Georgia", across five kinds of business and twenty U.S. cities. We asked each one on chatgpt.com, logged out, three times, a few minutes apart, and pulled the businesses named out of each answer.
Two answers to the same question shared, on average, about half their names. Measured as the overlap between the two lists (the number of shared names divided by the number of distinct names across both), the figure is 0.57. The business named first was the same in 65% of pairs. Some questions were steadier than others: property managers and home inspectors came back about two thirds the same, while "best realtor" and mortgage lenders came back half the same.
Waiting doesn't change this picture. A capture taken five hours later shared 0.57 of its names with the morning's, and one taken the next afternoon 0.55. The lists do drift, but slowly.
The practical reading is one we have made before and will keep making: a single answer is a sample, not a fact. If a report shows you one screenshot, that one answer is not representative of every single answer the AI engine could have given.
Now ask the API
With the portal's own consistency as the ceiling, we built the best script we could. The model was the one OpenAI lists as the default for free accounts, with the vendor's hosted web-search tool attached, and we then added, one setting at a time, the things a careful engineer would add in order to look more like the product: a U.S. location, a ChatGPT-style set of instructions, and forcing the search to run. We also varied search depth, reasoning effort, and the model itself, from the smallest to the flagship.
The result is easy to summarise. Without a search tool, the API's answers shared almost nothing with the portal's (0.08), which confirms that none of this comes from the model's memory. With the search tool attached and nothing else, agreement rose to 0.28. And then most subsequent changes we made had little effect. Location, instructions, forced search, search depth, and the flagship model at twenty-five times the price all landed within a hair of 0.28. The only setting that helped at all was turning reasoning effort down to its lowest value, which is also the cheapest, and it helped by three hundredths.
So a script reaches about half of the portal's own agreement with itself (0.28 against 0.57), on the first step, and stays there. That is not nothing: in 93% of cases the script's list and the portal's list shared at least one name, and of the businesses the portal named consistently (in at least two of three asks), three runs of the script reached 68%. But it is only a proxy for the real thing. The script is drawing from a neighbouring distribution: the same sources, the same kind of list, and a different sample.
Why the gap is where it is
The script was slightly less consistent with itself than the portal was: three API runs of the same question agreed with each other at 0.39, against the portal's 0.57. The reason is visible in the search results the API hands back. For the same question on two runs, the pages OpenAI's search tool returned overlapped at only 0.38. Most of the variation is in which pages come back, not in what the model does with them, and that is a part of the system no API setting controls.
What we suggest
- Treat any single AI answer about your business, from any source, as one draw. Three asks is the minimum before you believe a name is present or absent, and the honest unit is "named in at least two of three".
- If you are building or buying a tracker, ask which surface it measures. For ChatGPT, a script that calls the API with web search is a reasonable proxy for which businesses tend to be in the conversation and a poor proxy for what a given user saw. Capturing the actual product, through a browser, is more faithful, and in our hands it was also cheaper per answer.
- Do not pay for a bigger model to make a reconstruction more faithful. It does not help, because the gap is in the retrieval, not the reasoning.
In the next post we turn to Claude, where the gap is of a different kind entirely: the Claude you reach through the API does not search the same web as claude.ai does.