Where Does a ChatGPT Answer Come From? Cache, Index, and Live Fetch in OpenAI's Search Stack, Measured from Both Sides

1. Introduction

A widening share of the questions that once began at a search box now begin in a chat window. OpenAI reports that ChatGPT handles roughly 2.5 billion prompts per day (Axios, via TechCrunch, 2025) - already around a sixth of Google's estimated daily query volume, and it doubled in the seven months prior; Gartner projected a 25% decline in traditional search-engine volume by 2026 as chat assistants absorb it (Gartner, 2024). The commercial consequence is that for a small business, "how does ChatGPT describe me" is becoming the same kind of question "how do I rank on Google" was twenty years ago. The tooling, however, is not comparable. Google publishes an extensive webmaster program - documentation, diagnostics, per-URL index inspection, sitemap submission - through Search Central and Search Console. OpenAI publishes a one-page list of its crawlers and their IP ranges. There is no coverage report, no index-status tool, and no submission mechanism. A site owner who wants to know where they stand must measure it themselves, and it is not obvious how.

The difficulty is that "being in ChatGPT" is not one property. The answer a user sees is composed from up to four stores: the model's parametric memory laid down in pre-training, a fetch cache of recently retrieved URLs, a search index that answers queries, and the live web fetched on demand. The architecture is recognizable as the retrieval-augmented design of the research literature (Guu et al., 2020; Lewis et al., 2020), with OpenAI's WebGPT as its most direct published ancestor (Nakano et al., 2021), layered over the crawl–index–serve pattern of classical web search (Brin and Page, 1998). From the outside, however, the layers are indistinguishable, and popular diagnostics conflate them. A widely shared technique for "checking whether your site is in OpenAI's index" (llmrefs, 2026) in fact reads a different layer entirely, as we show in §4.2. Advice built on the conflation leads owners to fix the wrong problem.

In order to separate the layers cleanly, we measure from both sides at once. We sample ten production websites (nine for Homiere users whose business identities we withhold, plus our own brand site), so we can read every request that reaches them, including the requests of OpenAI's crawlers, verified against OpenAI's published IP ranges rather than trusted from user-agent strings. Simultaneously, we probe the system through OpenAI's Responses API, whose search tool exposes a trace of every query it runs and every page it opens. When the trace reports a page-open and the origin logs show no request arrived, the discrepancy is not noise; it is a measurement of which store served the content. Figure 1 shows the system as we model it, with the two observability planes marked.

We ask four questions. First, which layer answers which kind of question - and in particular, is anything about these businesses in the model's weights? Second, what populates the cache and the index respectively, and are they the same thing? Third, does Bing coverage predict ChatGPT-search visibility, as commonly assumed, or does OpenAI's own crawler decide it? Fourth, when a site is in the index, is it thereby retrievable - and when retrieval fails, what does the user actually see? Sections 4.1–4.6 answer these in order; §5 draws the practical map, and §6 states the limits of a ten-site, one-window study of a proprietary system.

2. Related Work

The layered architecture. The crawl–index–serve decomposition and its economics are classical (Brin and Page, 1998; the crawling literature is surveyed by Olston and Najork, 2010, whose treatment of selective crawl scheduling anticipates the admission behavior we observe in §4.3). Retrieval augmentation moved the pattern inside the language-model loop (Guu et al., 2020; Lewis et al., 2020), and search-augmented assistants made the model itself the author of the queries (Komeili, Shuster, and Weston, 2022; Nakano et al., 2021). That authorship matters for our results: retrievability in §4.4 is conditioned on queries the model writes, not queries the user writes. Vu et al. (2024) quantify how parametric knowledge fails on fresh and long-tail facts, the regime every small business occupies and the reason the retrieval layers dominate here.

Generative search engines and their measurement. Liu, Zhang, and Liang (2023) audited citation behavior in deployed generative search engines and found fluency systematically outrunning verifiability; Menick et al. (2022) showed that citing is a trained behavior layered onto retrieval rather than a transparent window into it. The optimization literature (Aggarwal et al., 2024) studies how content features shift a source's share of a generated answer with retrieval held fixed. Our own prior work took the same controlled-retrieval approach on Claude, measuring when the model answers from memory versus from evidence and how content features reorder a fixed result set (Dimitriou, 2026). The present study is complementary in method: instead of controlling retrieval and observing the model, we hold the production retrieval stack fixed and observe it, from the API trace on one side and from verified origin logs on the other. We are not aware of prior published work with simultaneous instrumentation of both planes.

Query taxonomy and crawler governance. Broder's classical taxonomy separates navigational from informational intent (Broder, 2002); §4.4 shows the production retrieval layer effectively does not serve the navigational class. Crawler behavior is governed by the Robots Exclusion Protocol (RFC 9309); OpenAI documents three crawlers with distinct roles (OpenAI, 2026), a separation our logs confirm is real and load-bearing.

3. Instrument and Methods

3.1 The set and the origin-side instrument

The origin side is a set of ten production websites we operate: nine for Homiere customers - (small businesses in the real estate vertical), anonymized throughout as Sites A–I - and our own brand site, disclosed as ours. All ten are served through the same edge infrastructure, where a middleware layer logs every request: timestamp, host, path, user-agent, and the client address as seen by the CDN. Identity is never assumed from the incoming user-agent string. Requests claiming to be OpenAI crawlers are verified against OpenAI's published IP ranges, and the verification is not a formality: in a 13-day window our own brand site received 67 requests wearing the OAI-SearchBot user-agent from addresses outside OpenAI's ranges - vulnerability scanners probing for exposed credential files - against 4 genuine visits. Any log analysis that trusts user-agents will overcount crawling by an order of magnitude on some hosts.

3.2 The API-side instrument

The API side is OpenAI's Responses API (model gpt-5.1) with the hosted web_search tool, probed between July 10 and July 30, 2026. The tool's output items record each search action with its model-written query and each open_page action with its URL, so fetching versus answering-from- results is directly observable per trial. The tool accepts a documented parameter, external_web_access: with the value false the model may use only content OpenAI has already stored; with true (the default) it may fetch live. Raw per-trial records - prompt, trace, sources, citations, output text - are retained for every trial. A parallel arm ran the same probes through the consumer ChatGPT interface (anonymous sessions, fresh state per trial) and agreed with the API arm on every mechanic reported here; where an arm-specific caveat matters we note it.

3.3 The probe ladder

Each probe is a prompt engineered so that, given the two-sided instrumentation, only one layer can account for a positive result. Table 1 lists the ladder.

3.4 Scoring, aggregation, and controls

Single probes are unreliable in both directions: the same fact one model-written query surfaces, a slightly different query misses, and we observed this within a single site on the same day. We therefore score retrievability per site as the count of no-fetch query shapes (P1, P2, P2b, P4) whose results contain the site's own URLs or answer its needle, and we treat any single negative as uninformative. The exact-phrase probe PX carries an asymmetric inference rule that its own controls forced on us: run against unique phrases from sites already proven indexed, it returned "no match" for three of four positive controls - the retrieval layer does not do literal phrase matching reliably - so a PX hit is decisive evidence of membership while a PX miss is evidence of nothing. Negative controls include an invented phrase (correctly returned no match) and, for each site, a question about a service the business does not offer; we additionally flag any trial in which the model asserts specifics that contradict the site's ground truth, which we know exactly because we operate the sites. Where we state a correspondence between crawling and index membership, crawl counts include only IP-verified requests, robots.txt polls and favicon requests excluded.

4. Results

4.1 Parametric memory: empty for every business tested

Closed-book probes (P0) drew a uniform result across all ten sites: the model declines, stating it does not know the business, with no leaked specifics. None of these entities is usably present in the weights, which matches the fresh-and-long-tail failure regime of parametric knowledge (Vu et al., 2024) and the measured-familiarity TAIL tier of our earlier study (Dimitriou, 2026), where unknown entities were refused in 91–100% of no-search trials. The consequence frames everything below: for businesses like these - which is to say, for most businesses - every word an answer engine says is retrieval, and the question is only which store it retrieves from.

4.2 The fetch cache is enforced, broad, and not the index

We ran the cache probe (PC) in three arms across the seven sites then in the ladder: cache-only, live, and cache-only again twelve minutes later. Table 2 shows the design and the outcome; the origin logs decide what served each answer.

Three properties follow. First, the parameter is real and enforced: zero origin traffic was observed in the cache arms, while exactly one fetch per site was observed in the live arm. Second, the cache is populated by ordinary fetch traffic and persists: two sites had never received a verified visit from OpenAI's indexing crawler, yet their homepages were readable from cache roughly 35 hours after our own earlier probes had caused them to be fetched. A user-triggered fetch is enough to plant a page. Third - the substantive point - the cache is much broader than the index: every site summarized accurately from cache, including the sites that §4.3 shows are absent from search. The tool trace, we note, records an open_page action even when the content is served from cache, so the trace alone cannot distinguish the two; only the parameter setting or the origin logs can. The popular cache-check technique (llmrefs, 2026) therefore measures layer 2, not layer 3: a page can be perfectly readable by this test and invisible to every user who searches rather than pastes a URL.

4.3 Index membership follows OpenAI's own crawler, not Bing

A common assumption is that ChatGPT search leverages Bing, so Bing coverage should predict ChatGPT visibility. We hold webmaster-tools data for the set of websites and tested this directly, and the result is clear: Bing serving is neither sufficient nor apparently necessary. Three sites with demonstrable Bing serving - real impressions and real clicks in the measurement window - scored zero or near-zero on every retrievability probe, while the two sites with the weakest Bing footprint were among the most richly retrievable. Whatever role Bing plays in discovery, membership in the index that answers ChatGPT's searches is decided elsewhere.

The origin logs give us a way to see where that decision is being made. OpenAI operates a dedicated indexing crawler, OAI-SearchBot, documented as distinct from both its training crawler and its user-triggered fetcher (OpenAI, 2026); our logs confirm the separation is real. Its behavior is strikingly selective: it polls robots.txt almost daily on every site, but reads actual pages on only some of them - and the correspondence between verified page-reads and index membership is nearly exact (Figure 2). Every site whose pages the crawler genuinely read is in the index; the two sites with no verified content-reads are the two absent from it. The correspondence held prospectively: one site had been fully crawled but never probed, so before probing it we predicted rich membership - and its needle question (a specific past transaction price) was answered from snippets, with the site's own pages among the sources and no fetch at all.

4.4 Presence is not retrievability

Site G is the most heavily crawled site in the set - 67 verified indexing-crawler fetches in 13 days - and it never appeared in any result for any natural question about the business, across every query shape, on repeated runs, including runs with the tool's search_context_size raised to its maximum. For weeks we classified it as not indexed. The classification was wrong. The site's homepage carries a distinctive form of words that appears nowhere else on the web, and a zero-competition exact-phrase query (PX) retrieved the site immediately - correct title, correct URL, from the index, with no fetch. The page was present the whole time; for every realistic query it was being outranked by the large listing aggregators that also describe the business. Membership and rank are separate properties, and only rank pays. It is worth noting which sites win their own entity queries in our set: the retrievable sites tend to be those whose third-party coverage is thin or inconsistent, while the two that lose are covered heavily and accurately by aggregators. Strong aggregator presence, often treated as free distribution, appears here as competition - an observation consistent with the finding that what a generative engine retrieves, cites, and actually uses are only loosely coupled (Liu, Zhang, and Liang, 2023; Menick et al., 2022), and with content features shifting share-of-answer independently of rank (Aggarwal et al., 2024).

Retrievability is also strongly query-shape-dependent, in a direction practitioners should know: the index does not serve navigational queries (Broder, 2002). A bare domain name as the query returned own-site results for at most three of ten sites, inconsistently across runs; site: enumeration returned nothing for any of the seven sites we ran it against, including ones proven present minutes earlier. Retrieval matches model-written query text (Komeili, Shuster, and Weston, 2022) against page content; asking by URL teaches you nothing about the index, and neither does any single negative probe (§3.4).

4.5 When retrieval fails, the model does not say so

From the user's chair none of this machinery is visible; what is visible is the answer, and in the failure cases the answers were consistently misleading in the same direction. Asked about content on an unindexed site, the model reported that the site "appears to be offline or not publicly accessible" - seconds before a direct fetch of the same site succeeded. Asked about the businesses behind unindexed sites, it guessed plausible-sounding domains that do not belong to them, opened strangers' pages, and confidently described same-named strangers: one subject was described, across repeated runs, by way of three different wrong people sharing the name, each with different incorrect biographical details. The model essentially never says "this business has a website I cannot see." It narrates index absence as a defect of the business - the site is down, the person is not findable, someone else answers to that name. We regard this as the single most practical reason for a small business to care about index membership: the alternative to being retrievable is not silence but misattribution.

4.6 What correlates with being crawled

If crawling gates the index, the operative question is what attracts the crawler. We computed per-site content and structure features - crawlable word count, page count, lexical diversity, the share of pages that are syndicated listing content, the presence of original articles, and domain age - and rank-correlated them against verified crawl volume and retrievability. At a small sample size of n = 10 these are directional estimates, reported in that spirit (Table 3, Figure 3).

Three patterns are worth stating. First, crawl volume tracks content mass and domain age, which is roughly how a selective crawler would rationally allocate budget (Olston and Najork, 2010). Second, features appear to act mostly through crawling: once a site is read at all, membership follows, so the feature story is largely a crawl-attraction story. Third, and most tentatively, retrievability leans away from syndicated content: the share of a site consisting of syndicated content (for example MLS listing data in the case of a real estat agent/brokerage) - content that exists in identical form on large portals - correlates negatively with winning queries, while original articles lean positive. This is consistent with the aggregator dynamic of §4.4: syndicated pages add bulk without adding any reason to retrieve you. The honest residual is Site H: an established domain, mid-pack in content, indexed and served by conventional search engines, whose robots.txt the crawler polls almost daily - and whose pages it has never once read in our verified window. Whatever admission logic sits in front of the crawl scheduler, it is not fully captured by size, age, or any content measure we computed.

5. Discussion

The four layers reduce, for a site owner, to four different problems that masquerade as one.

1. A cache check is not an index check. If a diagnostic shows ChatGPT "reading" your site under cache-only conditions, you have learned that your URL was fetched recently, by anyone, for any reason - including your own testing. Discovery runs through the index, and the two stores disagree on four of our ten sites.
2. The index is earned from OpenAI's own crawler, and its visits are observable. In our data, verified OAI-SearchBot page-reads are the single best indicator of membership, and their sustained absence is the clearest signal of invisibility - regardless of Bing coverage. A site owner with server logs owns a private, near-real-time admission monitor that no external tool currently provides; the one discipline required is IP verification, without which spoofed traffic will manufacture false comfort.
3. If you are crawled and still absent from answers, the problem is rank, not membership. The competition is the aggregators that also describe you, the referee is a language model reading snippets retrieved for queries it wrote itself, and the levers are the classical authority ones - consistent naming, consolidated profiles, original content the aggregators do not carry - plus the content-side levers our earlier work measured directly.
4. Guard the entity, not only the site. Same-name confusion and stale third-party profiles do not merely dilute an unretrievable business; they substitute for it, confidently. The failure narration of §4.5 means a prospect who asks about you and gets "their site appears to be offline," or a stranger's biography, has no cue that anything went wrong.

Two broader observations follow. The first concerns measurement: the two sides of our instrument disagreed often enough - traces reporting opens that never arrived, user-agents claiming identities their addresses disproved, "not found" verdicts on content that was provably present - that we would distrust any conclusion in this space drawn from one side alone. The second concerns the ecosystem: an index admitted by a selective, unannounced crawler, queried through model-written text, and narrated without provenance, concentrates real commercial consequence in mechanisms that site owners cannot see and that published guidance does not yet describe. The webmaster tooling gap of §1 is, in our view, the actionable half of that observation: the raw material for a coverage report - verified crawler visits - already flows through every site owner's logs.

6. Limitations

The sample size is ten sites, concentrated in one vertical (real estate and real estate adjacent) and our own brand site; the correlations of §4.6 are directional at this n, and the crawl–membership correspondence, while exact in our data and confirmed once prospectively, is a correspondence - the admission gate itself remains unobserved (Site H's exclusion is unexplained). The system under study is proprietary and changes without notice: results describe its observed behavior between July 10 and July 30, 2026, for one API model tier, with a consumer-interface arm run anonymously (the lightweight tier) rather than on a paid account. Retrievability scores rest on a fixed set of query shapes; the phrase-probe instrument can prove membership but not absence (§3.4), so "not in the index" verdicts are aggregates over shapes, not certainties - Site H's verdict in particular could in principle be deep outranking across every probe, though its zero verified crawls make absence the simpler account. Finally, the fetch-demand observations are timing evidence, not a randomized causal test, but they point in one direction: user-triggered live fetches can populate the cache even without verified OAI-SearchBot content visits.

Reproducibility. The probe harness, the exact prompts, the anonymized per-site data behind every figure and table, the correlation analysis, and the figure-generation code are released in the public research repository. Origin logs contain addresses and real hostnames and are not released; the log schema, the verification procedure against OpenAI's published ranges, and the queries we run against our own logs are documented in the repository so that any site operator can reproduce the origin-side instrument on their own infrastructure. Our sample is small and growing; site operators who would like to contribute their own verified crawl data to the next iteration of this study are invited to contact us.

References

1. Pranjal Aggarwal, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, Karthik Narasimhan, and Ameet Deshpande. 2024. GEO: Generative Engine Optimization. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD).
2. Sergey Brin and Lawrence Page. 1998. The Anatomy of a Large-Scale Hypertextual Web Search Engine. Computer Networks and ISDN Systems, 30(1–7), 107–117.
3. Andrei Broder. 2002. A taxonomy of web search. ACM SIGIR Forum, 36(2), 3–10.
4. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. REALM: Retrieval-Augmented Language Model Pre-Training. In Proceedings of the 37th International Conference on Machine Learning (ICML), PMLR 119:3929–3938.
5. Martin Koster, Gary Illyes, Henner Zeller, and Lizzi Sassman. 2022. Robots Exclusion Protocol. RFC 9309, IETF.
6. Mojtaba Komeili, Kurt Shuster, and Jason Weston. 2022. Internet-Augmented Dialogue Generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL), 8460–8478.
7. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Advances in Neural Information Processing Systems (NeurIPS).
8. Nelson F. Liu, Tianyi Zhang, and Percy Liang. 2023. Evaluating Verifiability in Generative Search Engines. In Findings of the Association for Computational Linguistics: EMNLP 2023, 7001–7025.
9. Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, and Nat McAleese. 2022. Teaching language models to support answers with verified quotes. arXiv:2203.11147.
10. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2021. WebGPT: Browser-assisted question-answering with human feedback. arXiv:2112.09332.
11. Christopher Olston and Marc Najork. 2010. Web Crawling. Foundations and Trends in Information Retrieval, 4(3), 175–246.
12. Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, and Thang Luong. 2024. FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation. In Findings of the Association for Computational Linguistics: ACL 2024, 13697–13720.