How Close Can an API Get to the Chat Window? Measuring the Gap Between Scripted Answer Engines and the Portals People Actually Use

1. Introduction

When a prospective home buyer asks ChatGPT or Claude who the best realtor in their city is, the answer they see is produced by a consumer product: a chat window with its own instructions, its own search tool, its own model selection, and its own usage limits. Almost nobody who studies or monitors these answers does so through that window. Research on answer engines, commercial "AI visibility" trackers, and our own earlier work frequently call the vendors' programming interfaces (APIs) from a script, attach the vendor's hosted web-search tool, and treat the result as a stand-in for what the user sees. The substitution is convenient, because an API call is cheap, repeatable, and returns structured data. However, it is also an assumption, which may not hold: if the script and the window disagree about which five businesses to name, then a visibility report built on the script is a report about a system nobody uses.

Our first study in this series examined how a model arbitrates between what it memorized and what it retrieves, using an instrument in which every search result was authored by us (Dimitriou, 2026a). We noted there that "the controlled search tool is a faithful shape of a real tool, not a real one; measuring the gap between this instrument and the production portals directly, rather than assuming it away, would require a separate portal-fidelity study." Here, we undertake that study. We use the term portal for the consumer chat product (chatgpt.com, claude.ai) and reconstruction for a script that calls the same vendor's API with its hosted web-search tool in order to imitate the portal. The question is how faithful a reconstruction can be made, which settings matter, and which gaps no setting can close.

Three things make the question harder than it sounds. First, the portal does not agree with itself: the same question asked twice returns different names, so a reconstruction cannot be expected to match the portal more closely than the portal matches its own next answer. Any measurement of fidelity therefore needs a ceiling, measured on the portal, and a floor, measured between systems that share nothing. Second, the portal is a moving target with undocumented parts: which model a free user is served, what instructions it runs under, where the search results come from, and whether the search is forced or optional are all product decisions that change without notice. Third, the comparison has to be made at the level that matters commercially. For a local business the unit of visibility is whether it is named, and where in the list, so we compare answers as ranked lists of named businesses rather than as text.

We focus on the casual, unpaid user, because that group represents a significantly large audience: chatgpt.com used without an account, and claude.ai on the free plan. OpenAI's help center names the model served to free accounts, claude.ai reports its model in the conversation record it keeps, and both vendors expose those models through their APIs, so the reconstruction can at least start from the same model. We then step through a ladder of settings, from no search tool at all, through the vendor's default search configuration, to the addition of location, a consumer-style system prompt, the portal's effort level, forced search, and a page-fetch tool, and we measure at each step how much closer the answers move to the portal. We run one-factor variations off the best step, and we substitute neighbouring models, in order to see which choices move the needle and which are inert. Finally, because both vendors expose the search queries their models wrote and the results those queries returned, we can locate the disagreement in the pipeline: in the query the model wrote, in the results the search backend returned, in the sources the answer cited, or in the businesses it named.

Our contributions are as follows. (1) A measurement design for portal fidelity with explicit ceilings and floors, applied to two vendors, five families of local-recommendation queries and twenty U.S. cities. (2) A configuration ladder with measured effects, showing which API settings bring a reconstruction closer to the portal and by how much. (3) A layer-by-layer decomposition that shows the reconstructions fail in different places: for OpenAI the gap is largely the ordinary run-to-run variability of a search-backed model, while for Anthropic the API's web-search tool and the claude.ai search tool retrieve different pages for the same query.

2. Related Work

Measuring visibility in generative engines. The optimization literature that followed Aggarwal et al. (2024) mostly measures answer share with the retrieved documents held fixed, and a growing body of commercial audits measures it on live systems. A recent critical survey of forty-five such studies finds low overlap between the sources different measurements report, high run-to-run variability, and what it calls fidelity gaps between the system measured and the system users see (Martinez, 2026). Schulte, Bleeker, and Kaufmann (2026) make the variability point directly: visibility in AI search is a distribution over repeated runs, not a value, and they excluded API data from their ChatGPT analysis precisely because the two surfaces are not interchangeable.

API proxies for the chat product. The closest prior work to ours is Jiang et al. (2026), who collected citation traces from authenticated ChatGPT sessions across several model generations and asked how well Google Search and OpenAI's Responses API predict which pages ChatGPT cites. They report that the API's own citations recover roughly a quarter of ChatGPT's cited hostnames and that ChatGPT's run-to-run citation overlap is itself only about 0.4 (Jaccard) at the hostname level. Our study differs in four ways: we compare the answers as ranked lists of named businesses rather than as sets of cited URLs, which is the unit of commercial consequence; we vary the API configuration systematically rather than fixing one; we study Claude alongside ChatGPT; and we target the logged-out and free tiers rather than authenticated sessions. The separation of citation selection from citation absorption proposed by Zhang, He, and Yao (2026) is related to our layer decomposition, which additionally includes the search queries the model wrote and the results the search backend returned.

Drift and personalization. The fact that a hosted model's behaviour changes between dated versions is well documented (Chen, Zaharia, and Zou, 2023). Our design treats the portal as a moving target and reports when each capture was made. Classical web search is personalized, with location and login state the main drivers and local-business queries among the most affected (Hannak et al., 2013). So we pass a location to the APIs for the same reason and test whether the operator's city, which the portal infers from the browser's network address, matters.

Comparing rankings. We use rank-biased overlap, a top-weighted similarity for lists that need not contain the same items (Webber, Moffat, and Zobel, 2010), alongside plain set overlap, because an answer engine's list is exactly the kind of indefinite ranking that measure was designed for.

3. Instrument and Methods

3.1 Queries

We built a grid of 100 queries by pairing each of five families of queries with each of twenty U.S. cities. The families are the local-recommendation questions a consumer asks when they need a real-estate professional: "best realtor in [city], [state]", "who are the top real estate agents in [city], [state]?", "best home inspector in [city], [state]", "best mortgage lender in [city], [state]", and "best property management company in [city], [state]". The cities span regions and sizes, from Providence and Savannah to Fresno and Tucson. Every query is phrased so that a useful answer is a ranked list of named businesses or people, which is what we compare. Some of the comparisons in §4 were run on all 100 queries and others on half of them, in order to limit the cost of the paid API calls. By arm we mean one configuration of the reconstruction, such as "search tool on, with a U.S. location", run over a set of queries and compared with the portal's answers to the same queries. The arms that change a single setting at a time, or swap in a different model (both are described in §3.3), were run on half of the grid. The half consists of every second city in our list, which keeps the cities spread across the country, with all five query families retained, so that each family still has ten of its twenty queries.

3.2 Method 1: the portals

chatgpt.com, logged out. We captured the logged-out ChatGPT web product with a scraping vendor's browser farm, by typing the prompt into chatgpt.com from a web proxy and awaiting the rendered answer. Each query was captured three times with the vendor's "web search" input flag on, which appends ChatGPT's own search hint to the page URL, and twice with the flag off, so that the product decides for itself whether to search; half the grid was captured once more five to six hours later and once more the following day, in order to measure how fast the product's own answers drift. As a check that the capture route is not itself a source of disagreement, we captured the half grid once more through a different web proxy with search enabled.

claude.ai, free plan. Anthropic's free tier requires an account, so we drove a real Chrome browser, signed in to a free-plan account from a residential address through the browser's debugging protocol. Each query was typed into a new incognito conversation (claude.ai's temporary-chat mode), so that the account's chat memory, which the product had enabled, could not colour the answers; when the answer finished we read the conversation record the application itself requests, which carries the model name, the effort setting, the temporary-chat flag, every tool call with its search query, every search result, and the citations attached to the answer text. The product reported itself as "Sonnet 5.5" at "Medium" effort with web search enabled on every capture.

3.3 Method 2: the reconstructions

OpenAI. Calls go to the Responses API with the hosted web_search tool. The model is gpt-5.6-luna, which OpenAI's help center lists as the default for Free and Go accounts at the time of the study; we include the tool's source list in the response so that the pages the search returned are recorded along with the citations. Anthropic. Calls go to the Messages API with the web_search_20260209 tool, called directly rather than through the tool's default "dynamic filtering" mode, except where that default is the arm under test. The model is claude-sonnet-5-5, which the free portal reported serving. Both vendors' search tools are priced per search in addition to tokens, and every request and response, including token usage and computed cost, is logged verbatim.

The configuration ladder adds one portal-like property at a time:

1. No search tool. The model answers from memory. This is the floor for a reconstruction and the condition our first study's closed-book probes used.
2. Web search, vendor defaults. The tool attached with no options.
3. + Location. A coarse user_location of country "US", the only location a logged-out scraping route can be said to have.
4. + Consumer system prompt. For Anthropic, the system prompt Anthropic publishes for claude.ai (Sonnet 5.5, October 7, 2026 revision), with the date filled in. OpenAI does not publish ChatGPT's instructions, so we wrote a short prompt in the same spirit (identity, date, a description of when to use the web tool for local and fresh information) and state plainly that it is an approximation.
5. + Portal effort level. For Anthropic, effort: medium, the value the portal reported; the API's default is high. For OpenAI, we instead test forcing the search tool, because the portal route we captured carries a search hint.
6. + Page fetch. For Anthropic, the web_fetch tool alongside search, since claude.ai can open pages.

Starting from a base configuration, which is the most portal-like setting we tested before adding forced search or page fetching (for OpenAI, the search tool with a U.S. location and a ChatGPT-like system prompt; for Anthropic, the same with the published claude.ai system prompt and the portal's medium effort), we run one-factor variations, each of which changes one setting and leaves the rest fixed: OpenAI's search_context_size (low, high), reasoning effort (none, low), the legacy web_search_preview tool, and a cache-only mode with live fetching disabled; Anthropic's search cap (max_uses 1 and 3), low effort, the operator's actual city as the location, a generic system prompt, and the settings our own production tracker uses. We also substitute models: gpt-6-luna, gpt-5.6-sol, gpt-5.5 and gpt-5.4-mini on the OpenAI side; Sonnet 5, Haiku 5.5 and Opus 5.5 on the Anthropic side. The anchor arms (vendor defaults, and the base configuration) were run two or three times per query so that the reconstruction's own run-to-run agreement can be measured; the one-factor variations and model substitutions ran once.

3.4 From answers to ranked lists

Each captured answer, from either method, is read by a small model (Claude Haiku 5.5) under a fixed instruction that extracts every business, team, or person the answer presents as an option, in order of appearance, with its affiliation, and that excludes directory and review sites that are merely cited as sources. Within each query, name variants are then clustered: "John Doe", "John Doe Team" and "John Doe - Example Realty" become one entity, while a team and its lead agent remain distinct, since merging on a shared surname would be wrong more often than right. The grouping is transitive over every name seen for the query across every arm (if one form matches a second and the second matches a third, all three are one entity), so the same identities are used on both sides of every comparison.

3.5 Agreement measures, ceilings, and floors

For two answers to the same query we compute: Jaccard overlap of the named sets (shared names divided by all names in either list); rank-biased overlap (RBO), which weights agreement near the top of the list more heavily (we use the setting under which the first ten positions carry most of the weight); whether the two lists share the same first name (top-1) and the overlap of their first three; and Jaccard overlap of the cited domains. Where both sides expose their search pipeline we also compute the token overlap of the first search query each model wrote (query similarity) and the overlap of the domains the search backend returned (result overlap). An arm's agreement with the portal is the mean of these measures over every (arm replicate, portal replicate) pair for a query, averaged over queries, with 95% bootstrap confidence intervals resampled over queries.

Two reference quantities make the numbers interpretable. The ceiling is the portal's best case run-to-run agreement with itself, computed over pairs of its own replicates. The floor is the agreement between the two vendors' portals on the same query, which is what two unrelated systems that read the same web produce. We quantify a reconstruction's agreement with the portal on the same scale, as overlap values. A reconstruction is as faithful as it can be when these values sit at the ceiling; it is uninformative when these values sit at the floor.

4. Results

4.1 The ceiling: a portal does not agree with itself

Before asking how close a reconstruction gets, it is necessary to know how close anything can get. Asked the same local-recommendation question three times, minutes apart, the logged-out ChatGPT product returned lists whose names overlapped by a Jaccard index of 0.57 (95% CI [0.53, 0.60]); the first-named business was the same in 65% of pairs, and the rank-biased overlap was 0.73. In other words, roughly one name in two carries over from one ask to the next, and the top of the list is somewhat steadier than the tail. The agreement varied by family (Table 1): the product was most consistent about property managers (0.68) and home inspectors (0.63), least consistent about mortgage lenders (0.48) and the plain "best realtor" question (0.50). These are the ceilings against which every reconstruction below is read.

The ceiling also holds over hours and days. A fourth capture of fifty of the queries, taken five to six hours after the first three, agreed with them at 0.57 (RBO 0.73), and a fifth, taken twenty hours later, at 0.55 [0.49, 0.61] (RBO 0.71, same first name 58%), against 0.60 among the three captures taken minutes apart on the same queries (§4.6). The lists drift, but slowly, and a reconstruction that sits at 0.28 is far below the portal's agreement with its own answer from the day before.

The free claude.ai product, asked each of fifty queries twice in separate incognito conversations, agreed with itself at the same level, 0.57 [0.49, 0.65] (RBO 0.71, same first name 66%), with the same spread across families: 0.79 for "top agents" and 0.73 for lenders down to 0.49 for "best realtor" and 0.27 for home inspectors, the family on which, as §4.5 shows, the product alternates between two different retrieval tools.

The capture route does not add disagreement of its own. Captures of the same product through a second scraping vendor agreed with the first vendor's captures at 0.53, and captures with the search hint switched off agreed with those with it on at 0.54, both within the portal's own replicate-to-replicate range. The product searched the web in essentially every answer whether or not it was hinted to (99% and 100% of answers carried citations), and named 4.7 businesses per answer on either route. At the other end of the scale, the two vendors' portals, which share nothing but the web, agreed with each other at 0.16 [0.13, 0.19] on the fifty queries both answered. That is the floor.

4.2 The ChatGPT ladder: the first step is the only step

Figure 1 (left) shows the ladder for OpenAI. With no search tool, the model answering from memory agrees with the portal at 0.08, below even the floor: whatever the portal says about local businesses, it does not come from the model's weights, consistent with our earlier finding that parametric memory held none of the businesses we tested (Dimitriou, 2026b). Attaching the vendor's hosted search tool with no other setting raises agreement to 0.275 [0.25, 0.30], a paired gain of +0.20 [0.16, 0.22]. Every subsequent step is inert: adding a U.S. location (0.277), a ChatGPT-like system prompt (0.264), both together (0.275), or forcing the search tool to run (0.273) produces paired differences whose confidence intervals all span zero. The reconstruction reaches about half the ceiling (0.275 / 0.567 = 0.49) at the first step and stays there.

Two further measurements say what kind of gap this is. First, the reconstruction is less consistent with itself than the portal is: three API runs of the same query agreed with each other at 0.39 (0.43 with forced search), against the portal's 0.57. The OpenAI search tool is a large part of that: the pages it returned for the same query on two runs overlapped at only 0.38, so most of the API's run-to-run variance is retrieval variance. Second, the disagreement is diffuse rather than categorical. In 93% of (API answer, portal answer) pairs the two lists shared at least one business; a business named in one API answer had a 37% chance of appearing in a given portal answer and roughly a 48% chance of appearing in at least one of three; and of the 4.3 businesses per query that the portal named in at least two of three asks (its "stable core"), three API runs reached 68%. A second capture route through the portal itself reached 88% of the same core with two runs. The API, in short, draws from a neighbouring distribution: the same sources, the same kind of list, the same businesses more often than not, but a different sample.

4.3 The Claude ladder: the retrieval layer is a different one

The right panel of Figure 1 shows results for the Claude reconstruction. Every Anthropic configuration that searches sits between 0.15 and 0.18 overlap with claude.ai, which is the floor: the API reconstruction of claude.ai agrees with claude.ai (0.16) exactly as well as chatgpt.com does (0.16, 95% CI [0.13, 0.19]), against a ceiling of 0.57, so the reconstruction reaches 29% of the ceiling where OpenAI's reached 49%. This value is insensitive to changes in how the reconstruction is built. Calling the tool directly rather than through the vendor's default dynamic filtering (+0.01), the U.S. location, the operator's actual city, the published claude.ai system prompt, the portal's "medium" effort, the search cap, a generic prompt, our own production settings, and three other models (Sonnet 5, Haiku 5.5, Opus 5.5) all leave it within two hundredths of the direct-search arm; adding the page-fetch tool is the only paired gain whose interval excludes zero, and it is +0.016 [0.002, 0.029]. Of the 7.1 businesses per query that claude.ai named in both of its replicates, two API runs reached 45%; on the OpenAI side the corresponding figure was 68%.

To see where the two surfaces diverge, we compared them at each stage of producing an answer (Figure 3): the search query the model wrote, the search results that came back, the pages the answer cited, and the businesses it named. For the same question, the model on both surfaces wrote nearly the same search query (token overlap 0.86 between surfaces, 0.96 between two API runs). The pages that came back were different: the search results the API received overlapped with those claude.ai received at 0.14, while two API runs overlapped with each other at 0.91 and two portal runs at 0.90. Each backend is nearly deterministic; they are deterministic about different pages. The two surfaces share a model and a query and do not share a search backend. Table 2 shows what each backend prefers. The API's tool returned a Yelp page in every one of the fifty answers and a U.S. News agent page in 38%; the portal's tool returned neither, not once, and leaned instead on Expertise.com, the referral directory Clever, RealTrends rankings, local ranking sites such as Doss, and, for the realtor queries, press-release wires (AccessNewswire and Newswire together appeared in 20% of portal answers and in no API answer). The answers provided by the platforms inherit the difference: the API cited Yelp in 40 of 50 answers, the portal in none, and the businesses named downstream overlap at 0.16. The disagreement is smallest where the two backends happen to share a dominant source, as for mortgage lenders (both lean on saferate.com; overlap 0.25) and property managers (0.21), and largest for the plain "best realtor" question (0.10), where the portal reads press releases and the API reads Yelp.

Two details of the portal's tool trace point the same way. Its search call carries a mode argument ("standard") that the API tool does not expose, its results arrive as "knowledge" items without the page-age field the API returns, and some of its URLs are in a raw ?p= form that the API's results never use, as if drawn from a differently canonicalized index. The portal also answers more briefly: 9.1 named businesses per answer against the API's 12.1 on the same queries, with the same mix of people and teams.

The portal has one more retrieval layer that the API lacks altogether. On 55% of the home inspector captures claude.ai did not search the web at all: it called a places_search tool that returns Google Places records (name, address, rating, review count), rendered them on a map, and wrote its answer from the ratings and reviews, phone numbers included. It did so on 5% of the property management and "best realtor" captures and never for lenders or "top agents". We return to this layer in §4.5, because chatgpt.com has one too.

Chat memory did not cause this. Our first twelve claude.ai captures were made before we switched to Claude's incognito mode, in ordinary chats with the account's memory feature enabled. Those answers agreed with the later incognito answers to the same queries at 0.46 (RBO 0.61, same first name 55%, search-result overlap 0.60), which is in the range of the portal's agreement with itself (0.57) and far above the API's agreement with either (0.10 on those queries). Thus, personalization due to being logged in with an account moved the answers no more than re-asking did. And the retrieval split, which is the difference between the pages the API's search tool returns and the pages the portal's search tool returns for the same query, is present in incognito captures that have no memory at all.

4.4 One-factor variations and model substitutions

Figure 2 shows what happens when we change one setting at a time from the base configuration, and when we swap in other models. On the OpenAI side only two variations moved the needle, and in opposite directions. Setting reasoning effort to none, the cheapest setting and the one closest to an "instant" consumer model, raised agreement to 0.30, a paired gain of +0.035 [0.009, 0.060] over the base configuration; substituting the small gpt-5.4-mini lowered it to 0.19, a loss of 0.083 [0.025, 0.136]. Everything else was inert within its interval: search context size low or high, reasoning effort low, the legacy web_search_preview tool, disabling live fetching so that the tool answers from OpenAI's cache, and the model substitutions gpt-6-luna (0.28), gpt-5.6-sol (0.28) and gpt-5.5 (0.29). The model substitutions are informative precisely because they are inert: the flagship model at twenty-five times the price of the free-tier model reconstructs the free portal no better than the free-tier model does, which is what one expects if the gap is in retrieval sampling rather than in the model's judgment.

One substitution deserves a separate note. gpt-6-luna, the newest and cheapest model, matched the base configuration on Jaccard (0.28) while naming far fewer businesses per answer (3.6 against 6.1). Fewer names means each one is more likely to be in the portal's list (pooled precision 0.61 against 0.40) but far fewer of the portal's names are reached (pooled recall 0.30 against 0.59, and 39% of the stable core against 68%). A reconstruction can therefore be tuned toward precision or toward recall by model choice without changing its overall overlap, and a tracker should know which it is buying.

4.5 What the answers look like

The two surfaces also differ in shape (Figure 5). The logged-out ChatGPT product named 4.7 businesses per answer; the OpenAI reconstruction named 6.1 to 6.5, and gpt-6-luna 3.6. The free claude.ai product named 9.1 and the Anthropic reconstruction 12.1. (We do not compare answer length in characters across capture routes, because the two scraping vendors render the same page to different lengths; the entity counts agree across routes, 4.5 and 4.7.) Per family, the OpenAI reconstruction tracked the portal best on property management (0.39) and worst on mortgage lenders (0.21), the same ordering as the portal's own self-agreement, so the hard families are hard for the same reason on both surfaces: the web offers many plausible candidates and no dominant list.

Both portals also have a layer the APIs do not: a local-places tool backed by business-profile data rather than web pages. chatgpt.com showed a map card of businesses on 55% of its answers overall, and on 70% of home inspector and 75% of property management answers against 27% of "top agents" answers (Figure 6); claude.ai reached for its places tool on 55% of inspector captures and almost never otherwise. What such a tool reads is ratings and reviews, not web pages, and neither vendor's API exposes anything like it. For the categories where it fires, a sizeable part of what the user sees comes from a store a reconstruction cannot reach, which is a second, separate reason the inspector family reconstructs poorly on both vendors, and the reason claude.ai is least consistent with itself on that family (0.27): an answer written from Google ratings and an answer written from web pages name different businesses.

4.6 How fast the ceiling itself moves

We asked how fast the ceiling itself moves, since a weekly tracker compares answers days apart, not minutes apart. On the fifty-query half grid, the three morning captures agreed with each other at 0.60 [0.55, 0.66]; a capture five to six hours later agreed with them at 0.57 [0.51, 0.63], and a capture the following afternoon at 0.55 [0.49, 0.61]. The decay is real but gentle, about a twentieth of the overlap per day at this horizon, and the first-named business stayed the same in 58% of next-day pairs against 65% within the batch. A reconstruction's agreement with the portal (0.28) is therefore not a matter of timing: the portal's answer from yesterday predicts today's answer twice as well as the best API configuration does.

5. Discussion

The question we set out to answer has two parts, and they have different answers. Can a reconstruction be engineered to match the portal? It was easier for ChatGPT and harder for Claude. For ChatGPT, the reconstruction reached about half of the portal's own best case run-to-run agreement (0.28 against 0.57), and no change we made, including the model itself, moved it further. For Claude, the reconstruction reached 0.16 against the same ceiling, which is the level at which the other vendor's portal agrees with it, because the API's search tool and the portal's search tool retrieve different pages for the same query (§4.3). What is the residual gap made of? For ChatGPT it is sampling: the same sources, the same kind of list, a different draw, with the API's own retrieval varying more from run to run than the portal's. For Claude it is a different retrieval layer, and so a different system.

Three implications follow for anyone who measures these systems, including ourselves.

1. Name the surface. A result obtained through an API with web search describes that API. For ChatGPT it is a usable proxy for which businesses tend to be in the conversation (three runs reach about two thirds of the portal's stable core) and a poorer proxy for what a given user saw in one query. For Claude it is a weaker proxy, reaching 45% of the names the portal gives consistently. Reports built on it should be labelled accordingly.
2. Report distributions, and report the ceiling. The portal's agreement with itself (0.57 on these queries, lower for agents and lenders) bounds every fidelity claim, and a reconstruction that is less self-consistent than the portal, as OpenAI's is, cannot be made faithful by averaging alone. Schulte, Bleeker, and Kaufmann (2026) argued for distributions over single runs; we add that the distribution's width should be measured on the portal, not on the proxy.
3. Spend on capture, not on models. The inert model substitutions are the clearest result in the study. A flagship model does not reconstruct a free portal better than the free-tier model does, because the disagreement is upstream of the model. Where the portal can be captured directly it should be, and in our hands that capture was also an order of magnitude cheaper per answer than the API call it would replace.

The result also answers the question we left open in our first study. The controlled search tool used there was a faithful shape of a real tool; what the present study shows is that even the real tool, the vendor's own hosted search, is not the portal's tool, and that the difference can be large. In practice the model matters less than the retrieval behind it: the information each surface retrieves varies more than the model does. Findings about how a model weighs retrieved evidence therefore carry over across these surfaces, but findings about which evidence is retrieved do not, and should be re-measured on the surface they are claimed for.

6. Limitations

The portals are moving targets, and the numbers describe them between October 20 and 21, 2026, for one query family (local recommendations in the real-estate vertical), twenty cities, and the free and logged-out tiers. The claude.ai arm covers half the grid, captured from one account in one city, and was rate-limited to a few windows of a dozen messages each; its ceiling is estimated on fewer replicates than ChatGPT's. Entity extraction and clustering are automated and were spot-checked rather than exhaustively audited; errors would add noise symmetrically to the ceiling and to the reconstructions, lowering both. Finally, the same-day and next-day portal replicates (§4.6) cover one day on half the grid; the portal's agreement with itself over weeks, which is the ceiling a weekly tracker actually faces, deserves its own study.

References

1. Pranjal Aggarwal, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, Karthik Narasimhan, and Ameet Deshpande. 2024. GEO: Generative Engine Optimization. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD).
2. Lingjiao Chen, Matei Zaharia, and James Zou. 2023. How Is ChatGPT's Behavior Changing over Time?. arXiv:2307.09009.
3. Chris Dimitriou. 2026a. When Does the Model Look It Up? Parametric Knowledge, Controlled Retrieval, and the Mechanics of AI Answer Engines. Homiere Research.
4. Chris Dimitriou. 2026b. Where Does a ChatGPT Answer Come From? Cache, Index, and Live Fetch in OpenAI's Search Stack, Measured from Both Sides. Homiere Research.
5. Aniko Hannak, Piotr Sapiezynski, Arash Molavi Kakhki, Balachander Krishnamurthy, David Lazer, Alan Mislove, and Christo Wilson. 2013. Measuring Personalization of Web Search. In Proceedings of the 22nd International World Wide Web Conference (WWW).
6. Yunxuan Jiang, Shiqi Zhang, Anqi Xing, Xiangyu Chang, and Xiaokui Xiao. 2026. Understanding ChatGPT Citations: Evidence from Production Web Search Across Model Generations. arXiv:2610.15382.
7. Olivier Martinez. 2026. Optimizing Visibility in Generative Engines: A Critical Survey of Generative Engine Optimization (2023-2026). arXiv:2607.14035.
8. Julius Schulte, Malte Bleeker, and Philipp Kaufmann. 2026. Don't Measure Once: Measuring Visibility in AI Search (GEO). arXiv:2604.07585.
9. William Webber, Alistair Moffat, and Justin Zobel. 2010. A Similarity Measure for Indefinite Rankings. ACM Transactions on Information Systems, 28(4), 20:1-20:38.
10. Kai Zhang, Xinyue He, and Jingang Yao. 2026. From Citation Selection to Citation Absorption: A Measurement Framework for Generative Engine Optimization Across AI Search Platforms. arXiv:2604.25707.