ChatGPT cited Reddit in 2 of 100 answers. The clue was in its searches

Last week, <a href="/articles/reddit-chatgpt-citation-share-source-concentration-risk">AEO Updates reported that Reddit’s citation share had fallen sharply in Promptwatch’s observed ChatGPT Search dataset</a>. The article treated the movement as a warning about source concentration, not proof of an OpenAI policy change. A new controlled experiment from OrganikPI does not settle the cause of that broader decline. It does show one way a source can disappear before citation selection even begins.[1] [2]

The experiment points to the retrieval layer. In OrganikPI’s test, ChatGPT rarely searched for Reddit unless the prompt expressed a need for first-hand experience. When a paired set of prompts added one sentence asking what real users said, Reddit’s citation share rose from 0.8% to 14.3%.[1]

That result changes the practical AEO question. It is not only which sources an AI system is willing to cite. It is also which sources the system’s generated searches make eligible to be cited.

<h2>The same 100 prompts produced five different Reddit profiles</h2>

OrganikPI ran the same fixed set of 100 prompts through ChatGPT, Google AI Mode, Gemini, Perplexity and Copilot on August 23, using a US geography. None of the prompts named Reddit. Fifty were designed to seek lived experience or subjective judgement, while fifty asked factual, procedural or mechanistic questions.[1]

Reddit represented 10.7% of Perplexity’s citations and appeared in 47 of its 100 answers. Google AI Mode gave Reddit an 8.1% citation share and used it in 50 answers. Gemini’s share was 6.9%, with Reddit appearing in 15 answers. ChatGPT cited Reddit in only two answers, representing 0.9% of its 1,221 citations. Copilot cited it in one answer, or 0.3% of its citations.[1]

The comparison makes one point unusually clear. Reddit had not become unusable to generative search as a category. The same prompt set produced materially different source mixes across the five AI systems.

The difference between prompt purposes was also visible. In Google AI Mode, Reddit represented 13.5% of citations for the experience-seeking half of the prompt set and 2.8% for the factual half. In Perplexity, the corresponding shares were 14.6% and 6.7%.[1]

Citation share was therefore shaped by both the AI system and the purpose composition of the prompt set.

<h2>The more revealing evidence was inside ChatGPT’s searches</h2>

OpenAI documents that ChatGPT Search typically rewrites a user’s question into one or more targeted queries. It may issue additional, more specific queries after reviewing initial results.[4] This process is often described as query fan-out.

OrganikPI could see ChatGPT’s generated search subqueries for 46 of the 100 ChatGPT responses. Across that subset, ChatGPT included the word “Reddit” in a subquery once.[1]

When the visible subquery named Reddit, 21.3% of the resulting citations pointed to Reddit. When it did not, Reddit represented 0.1% of citations, or one Reddit URL among 1,002 citations.[1]

This does not reveal every internal retrieval decision. Subqueries were visible for less than half the ChatGPT observations, and an interface-visible search query is not a complete map of the system. But within the observable subset, whether ChatGPT went looking for Reddit strongly predicted whether Reddit appeared in the citations.

<h2>One sentence changed the source set</h2>

OrganikPI then ran a paired ChatGPT experiment with 25 prompts. Each was submitted as written and with the sentence “What do real users say about it?” appended. Neither version named Reddit.[1]

The original versions produced 396 citations. Three pointed to Reddit, giving it a 0.8% share, and Reddit appeared in three of 25 answers. The experience-seeking versions produced 488 citations. Seventy pointed to Reddit, a 14.3% share, and Reddit appeared in 19 of 25 answers.[1]

The change did not cause ChatGPT to search the web more often. Web search occurred in 12 of 25 original prompts and 11 of 25 experience-seeking prompts. What changed was what ChatGPT searched for. None of the 12 visible original subqueries named Reddit. Ten of the 11 visible subqueries generated for the experience-seeking versions did.[1]

Sixteen prompt pairs moved from no Reddit citation to at least one, while none moved in the opposite direction. OrganikPI reports a two-sided McNemar exact-test value of approximately 0.00003 for those discordant pairs.[1]

The experiment is small, but it is more informative than a loose collection of screenshots. It changes one element of the prompt, holds the paired structure constant and observes a large directional change in generated searches and citations.

<h2>“De-defaulted” is a useful hypothesis, not an OpenAI disclosure</h2>

OrganikPI describes Reddit as having been “de-defaulted” rather than demoted. In that interpretation, ChatGPT is not necessarily retrieving Reddit and assigning it less authority. Reddit may be absent because the generated searches do not make it part of the candidate source set unless the information need calls for first-hand experience.[1]

The phrase is useful because it separates retrieval from citation selection. It should not be mistaken for a confirmed account of OpenAI’s internal logic.

Promptwatch observed a separate break on August 8. In its dataset, the share of ChatGPT Search fanout queries using the `site:` operator rose from 0.37% to 16.8%, while average fanouts per response rose from approximately 1.08 to 1.83.[3] The earlier AEO Updates article noted that this happened before Reddit’s citation share fell below 1% in Promptwatch’s series.[2]

The timing makes the observations worth studying together. It does not establish that the August 8 fan-out change caused the later Reddit decline. OpenAI’s public documentation confirms targeted query rewriting but does not announce an August 8 rollout, a Reddit-specific retrieval rule or an explanation for Promptwatch’s data.[4]

<h2>Citation share is partly a property of the prompt set</h2>

The larger measurement lesson extends beyond Reddit.

Suppose one visibility platform reports that Reddit represents 3% of citations while another reports 9%. It is tempting to decide that one number is wrong. But if one prompt set is weighted toward factual questions and the other contains more experience-seeking prompts, each figure may accurately describe its own Measurement Frame.

That is why prompt-set purpose and composition belong beside every AI Search measurement. A result needs a defined prompt set, a Prompt Taxonomy that captures dimensions such as Purpose, and a stable Measurement Frame covering the AI system, geography, time and other relevant conditions.

Changing the prompt set can change the metric even when the AI system has not changed. Conversely, an AI system can change its retrieval behaviour while the prompt set remains fixed. Without those distinctions, a movement in citation share can be attributed to the wrong part of the measurement chain.

<h2>This is a follow-up to the Reddit concentration warning, not a replacement</h2>

Promptwatch’s August series and OrganikPI’s experiment answer different questions.

Promptwatch observed a sharp change in Reddit’s share across a much larger, continuously monitored ChatGPT Search dataset. Its finding remains provisional because a collection-side issue cannot be completely excluded.[2] OrganikPI ran a smaller controlled test designed to compare AI systems and prompt purposes on a single day.[1]

The new experiment cannot reconstruct the complete cause of Promptwatch’s decline. It adds evidence that retrieval strategy and prompt purpose can materially affect whether Reddit enters the citation set. The connection is therefore mechanistic and explanatory, not a direct replication.

Together, the two stories make the concentration-risk lesson more specific. Source resilience is not only about appearing across several domains. It is also about producing evidence that can be found under several kinds of information need and through several retrieval paths.

<h2>What brands should do differently</h2>

The lesson is not to add the word Reddit to tracked prompts or to abandon Reddit because ChatGPT used it less in one test. The same OrganikPI prompt set found Reddit remained materially more visible in Perplexity, Google AI Mode and Gemini.[1]

Brands should separate measurement by AI system and Purpose rather than relying on a blended citation score. They should preserve a stable core prompt set for trend comparisons and document changes to the prompt universe. Experience-seeking, factual, comparison and recommendation prompts may activate different evidence environments, so each deserves explicit representation in the Prompt Taxonomy.

The retrieval layer also strengthens the case for an evidence portfolio. First-party pages, specialist publishers, datasets, reviews, community discussions and credible third-party coverage create different ways for an AI system to find support. No single source or content format should be assumed to remain a permanent default.

<h2>Important limitations</h2>

OrganikPI’s experiment used one run per prompt and AI-system cell, one US geography, one collection day and a deliberately balanced set of 50 factual and 50 experience-seeking prompts. Several comparisons contain small absolute citation counts. ChatGPT exposed search subqueries for 46 of the 100 cross-engine responses and roughly half the paired observations.[1]

The study’s absolute citation shares should not be compared directly with Promptwatch’s larger and differently composed prompt universe. The experiment demonstrates that prompt purpose can alter observed retrieval and citation behaviour under its controlled conditions. It does not prove OpenAI intent, establish the complete cause of Reddit’s August decline or show that every topic will produce the same effect.

<h2>AEO Updates Takeaway</h2>

The final citation is the visible endpoint of a longer process: information need, prompt, generated searches, candidate retrieval, evidence selection, answer and citation. If a source never enters the candidate set, downstream citation optimisation cannot help it.

The practical question is no longer only, “Which sources does this AI system cite?” It is also, “Which sources does this AI system go looking for when the prompt expresses this purpose?”

Citation share is not only a property of the AI system. It is also a property of the prompt set and the retrieval paths those prompts activate.

<h3>References</h3>

[1] <a href="https://organikpi.com/blog/geo-ai-search/chatgpt-reddit-citation-collapse/" target="_blank" rel="noopener noreferrer">OrganikPI: ChatGPT Cites Reddit on 2% of Answers. AI Mode Cites It on 50%</a>

[2] <a href="https://promptwatch.com/data/reddit-citations-are-dropping-in-chatgpt" target="_blank" rel="noopener noreferrer">Promptwatch: Reddit Citations Are Dropping in ChatGPT</a>

[3] <a href="https://promptwatch.com/data/chatgpt-site-operator-fanouts" target="_blank" rel="noopener noreferrer">Promptwatch: ChatGPT Search Now Uses the site:operator at Scale</a>

[4] <a href="https://help.openai.com/en/articles/9237897-chatgpt-search" target="_blank" rel="noopener noreferrer">OpenAI: Searching the web with ChatGPT</a>