This is the third of three posts on reproducing the answers that AI portals provide to questions asking to rank local businesses. The first measured how repeatable ChatGPT's local-business lists are and how well a script can stand in for them; the second looked at the same for Claude. This one turns the findings into a checklist. The full study is here: How Close Can an API Get to the Chat Window?
Agents ask us, often, to "check what ChatGPT says" about them. It is a reasonable request and a surprisingly easy one to answer badly. Over the past weeks we asked the two biggest assistants the same hundred local questions many times, through the chat windows and through the vendors' programming interfaces, and compared the lists of businesses they named. The numbers are in the study; the habits they support are below.
1. Ask at least three times, and count what repeats
Asked the same question three times a few minutes apart, chatgpt.com returned lists that shared about half their names. The business named first was the same only about two times in three. So if you search yourself once and do not appear, you have learned very little, and the same is true if you do appear.
The honest unit is not "named" but "named in at least two of three asks". In our data the average question had about four businesses in that stable group, out of roughly five named per answer. Those are the names the product is confident about; the rest rotate.
2. Know which surface you are looking at
There are at least three different systems that all get called "ChatGPT" or "Claude":
- The chat window a consumer uses (the portal), with the vendor's own instructions, its own search tool, and whatever model the vendor assigns to that plan.
- The vendor's API with its web-search tool attached, which is what nearly every monitoring tool and most research uses, because it is fast and returns structured data.
- The bare model with no search, which is what you get if someone forgets to attach the tool.
The third tells you nothing about local businesses: in our tests it agreed with the portal roughly as often as the other vendor's product did. The second is a reasonable proxy for ChatGPT or Claude in one specific sense: the businesses it names are the businesses the portal tends to name, and a few runs of it reach about two thirds of the portal's stable group for ChatGPT and under half for Claude. It is a poor proxy in another sense: a single API answer and a single portal answer overlap by only about a quarter, half of what two portal answers overlap by.
When a vendor shows you a report, ask which of the three it measured. If the answer is "the API", ask whether the search tool was on, and treat the result as a tendency, not a sighting.
3. Do not pay for a bigger model to get a truer picture
Substituting the flagship model for the free-tier one in our ChatGPT reconstruction, at roughly twenty-five times the price, changed the agreement with the portal by nothing. Adding location, instructions, forced search, or deeper search context changed it by nothing. The only setting that helped was the cheapest one, turning reasoning effort to its lowest value, and it helped by three hundredths. The gap between a script and the window is in which pages the search returns, which no API setting controls, not in how clever the model is. This perhaps makes sense, because AI platforms tend to offer a low-cost version of their product to anonymous users.
4. Expect some questions to be noisier than others
The portal was most consistent about property managers and home inspectors (about two thirds of names repeated) and least consistent about mortgage lenders and "best realtor" (about half). An agent will see more churn in their visibility than an inspector will, for the same underlying presence, because the web offers more plausible candidates and no single dominant list. Read your own trend over weeks, not days, and read it against the churn of your category.
5. Treat every number here as dated
Both products changed model defaults during the months we were working on this series, and neither announces what its search backend does. We report when each capture was made, and we will re-run the study as the products move. If you want to know what the assistants currently say about you, measured the way we describe above, let us know and we are happy to run it.