Researchers at the National University of Singapore and Xi’an Jiaotong University discovered that OpenAI’s API and Google search results partially matched the web pages cited by ChatGPT, according to findings published in an arXiv preprint on October 11, 2026. The study analyzed data collected between March 29 and July 29, 2026, across four versions of ChatGPT, revealing divergence between what users see in the web application and what external interfaces retrieve.
Tracking the Data Flow Inside ChatGPT
To capture how ChatGPT selects its sources, the researchers recorded the live data flow within their own signed-in sessions. This monitoring tracked the search queries executed by the platform, the resulting pages, and the final selections made by the model. The evaluation focused on 80 questions sourced from Search Arena, a public database of real user interactions with search-enabled AI models. The test set contained 40 English and 40 Chinese queries, spanning four distinct task types: recommendation, fact lookup, information synthesis, and analysis.
Each of the 80 prompts was executed 20 times using GPT-5.3 Instant and GPT-5.5 Instant. Earlier iterations, specifically GPT-5.2 Instant and GPT-5.4 Thinking, processed fewer questions because they were decommissioned before the data collection concluded. To evaluate the results externally, the authors retrieved up to five pages of organic Google results per question using SerpAPI, averaging about 43 results per question. Simultaneously, they submitted 20 queries per question to gpt-5.3-chat-latest and gpt-5.5 via OpenAI’s Responses API, with web search enabled, completing these requests within three days of running the tests on the ChatGPT web application.
Divergence Between Pages and Sites
The combined output of Google’s organic search results and OpenAI’s API matched 28.3% of the pages cited by GPT-5.3 Instant and 26.2% of the pages cited by GPT-5.5 Instant. When evaluated at the domain level rather than the exact URL, the overlap increased. The same external sources captured 50.3% of the sites cited by GPT-5.3 Instant and 48.7% of the sites cited by GPT-5.5 Instant, treating subdomains as separate entities.
Google search results alone accounted for 9.4% of cited pages and 26.1% of cited sites for GPT-5.3 Instant, alongside 6.4% of cited pages and 22.8% of cited sites for GPT-5.5 Instant. Meanwhile, the API’s entire source list accounted for more than that, showing 22.9% and 22.5% of all cited pages. However, the specific pages the API cited within its own direct answers showed a closer alignment to ChatGPT’s choices. GPT-5.5 Instant cited 23.3% of the pages generated by the API, while GPT-5.3 Instant matched 16.4%.
Query Rewriting and Inter-Run Variance
OpenAI’s documentation notes that when ChatGPT search relies on an external provider, it typically transforms an initial user prompt into one or more specialized search terms. In the recorded runs for GPT-5.3 Instant and GPT-5.5 Instant, the model never executed a search using the exact wording of the original question. By contrast, GPT-5.2 Instant retained the exact user phrasing in 5.5% of its runs.
Language translation during the search phase also created variance. Among the runs with visible searches for Chinese-language questions, GPT-5.3 Instant utilized English exclusively in 67.0% of the traces, while GPT-5.5 Instant did so in 69.6% of the traces.
Individual runs displayed notable inconsistency. For a given question, 44.5% of the distinct pages returned by GPT-5.5 Instant appeared in only one of the 20 test runs. Similarly, 40.8% of the distinct pages returned by GPT-5.3 Instant appeared in a single run. Overlap between runs was greater at the domain level than at the page level.
Monitoring Strategies for Developers and Auditors
Neither Google’s search results nor OpenAI’s API reproduced ChatGPT’s cited URLs exactly as they were cited, confirming that tracking tools relying purely on ranking APIs will capture a larger share of cited domains than individual webpages. The study’s authors recommend that developers and analysts monitor citations at both the webpage and domain levels, run identical prompts multiple times to account for run-to-run variance, and factor in that the model rewrites user prompts before executing searches.
Auditors must also verify whether comparison data stems from the API or the direct web application. Every setting evaluated in the paper predates the rollout of GPT-5.6 and GPT-6.
