AI search gives social science a new object of study

A household wants smaller portions, less food waste and a grocery bill that stays within budget. It asks an AI-search assistant how to adapt its shopping. What changes if the same question also mentions that one adult uses a prescribed GLP-1 medicine? Would the answer still focus on affordable choices, or introduce a different set of products and assumptions? Surveys and shopping diaries can investigate people's needs, while transaction records show what they buy. Controlled prompting can examine the guidance offered about those needs. The answer is a different object of study from the shopper.

That distinction gives AI search a role in social research beyond speeding up analysis or simulating respondents. A system that retrieves information, synthesises explanations and recommends options becomes part of the environment in which people make decisions. Researchers can investigate how it interprets a question, what it treats as relevant and whether its suggestions fit the circumstances supplied. Those observations complement evidence about people; they do not replace it. The shopping scenarios below are proposed research designs, not completed experiments or personalised dietary advice.

The answer is the object of study

Social scientists already study how institutions, media and digital systems represent problems and organise choices. Algorithm auditing also has an established methodological history: Sandvig and colleagues' paper, presented in May 2014, adapted established audit methods to algorithmic systems, including designs with controlled interactions. [1] AI search extends that research agenda to conversational guidance. The novelty is the object being examined, not the invention of content analysis or experimental comparison.

An answer can turn “How can this household spend less?” into advice about planning, product substitution, specialist support or purchasing something new. Researchers can examine which interpretation dominates, which options disappear and whose evidence supports the explanation. The same approach can investigate whether employment guidance recognises caring responsibilities or whether financial explanations accommodate irregular income. These are questions about how circumstances are represented, not just which brands appear.

The unit of observation might be a complete answer, a specific claim or a conversation that develops through follow-up questions. None is a person. The research question is not whether AI can pretend to be a person. It is how AI advises the person who asks. An answer alone cannot reveal the user's beliefs, the system's hidden reasoning or what anyone subsequently does.

What surveys, listening and search data establish

Human research establishes the context that makes an answer worth studying. A well-designed survey can measure experiences, attitudes and intentions in a defined population. Interviews and diaries reveal trade-offs and constraints that a questionnaire may miss. For the shopping example, these methods could investigate portion preferences, waste, price pressure and how responsibilities are shared. Transaction and operating records add purchases and availability within their coverage, although a basket does not explain its owner's motives.

Social listening captures language and narratives in the discussion collected. It can help identify questions about GLP-1 shopping or the meaning people attach to “support”, but visible discussion is not automatically representative. AAPOR's 2014 social-media public-opinion research report identified coverage, sampling and validation as central concerns. Its historical discussion supplies methodological cautions, not an evaluation of current AI-search systems. [2] Google Trends supplies another observation: sampled, aggregated search interest, normalised to a 0–100 index for the selected geography and period. It is not polling or absolute search volume, and Google excludes internal searches made by products including AI Mode and AI Overviews. [3]

Together, these sources can inform realistic research questions. Consented journey diaries or prompt records can establish what people actually ask assistants; a popular search term cannot establish that by itself. Controlled AI-answer studies then examine the guidance supplied under specified conditions. Their additional value is the opportunity to compare that guidance with human priorities and practical reality, without combining discussion, interest, recommendations and purchases into one ambiguous score.

Controlled prompting turns a demonstration into a research design

An isolated screenshot can illustrate a question but cannot establish a stable pattern. A controlled study begins with a hypothesis, a defined contrast and outcomes specified before collection. For the shopping example, the hypothesis is that disclosing prescribed GLP-1 use may change the products, evidence or explanations supplied even when the stated needs and budget remain constant. Two matched prompts make that contrast explicit.

Prompt A: treatment status unspecified

A two-adult household in Manchester has a weekly grocery budget of £80. The household wants smaller portions and less food waste. How could it adapt its grocery shopping while keeping costs down?

Prompt B: prescribed treatment use disclosed

A two-adult household in Manchester has a weekly grocery budget of £80. The household wants smaller portions and less food waste. One adult uses a prescribed GLP-1 medicine. How could it adapt its grocery shopping while keeping costs down?

The manipulated feature is disclosure, not appetite, household size or spending capacity. Prompt A does not describe a non-user; it leaves treatment status unspecified. Any measured difference concerns the system's response to disclosed context, not a biological effect of medication. A separate affordability comparison could retain prompt B and change only £80 to £50. Those budgets are researcher-selected test conditions, not estimates of typical or adequate spending. A more advanced factorial design could cross disclosure and budget to investigate whether their effects on answers interact.

Collect repeated samples in new conversations, interleaving or randomising the conditions within a comparable time window. Keep wording, intended market, language and search mode stable. Record the exact prompt, timestamp, visible platform settings, citations, refusals and available personalisation controls. OpenAI documents that relevant saved memories may influence search-query rewriting when memory is enabled, and approximate IP-derived location may inform local searches. A new chat alone therefore does not establish an unpersonalised baseline; that is a study-design inference, not a validated depersonalisation protocol. [5] Preserve complete answers, and distinguish an actual consumer-interface study from an API study. Pilot variability should inform repetition counts; no universal number makes a design adequate. Analyse follow-up conversations separately from fresh-session tests, and check realistic paraphrases of the same scenario: Zhuo and colleagues' ProSA research in Findings of EMNLP 2024 documents prompt sensitivity across tested LLM tasks and models, not a grocery-specific effect. Rephrasing the same question is a different contrast from adding treatment information. [4]

Look beyond inclusion to framing, feasibility and evidence

A useful coding guide would ask whether the answer retains the budget constraint, recognises portion and waste needs, and explains relevant trade-offs. Smaller packs can involve a different unit price from larger ones, so recommending them without a comparison may leave the household's question unresolved. Code ordinary shopping options separately from specialist products, and distinguish a passing mention, an endorsement and a warning. A specialist recommendation is not inherently inappropriate; an unchanged answer is not inherently inadequate. The test is relevance and support, rather than a result chosen in advance.

Verify material claims against dated evidence. Prices and availability need appropriate retailer records; treatment-related claims need qualified review and credible clinical sources. A citation must support the particular claim, not merely link to a page about the subject. Citation verification already has an empirical research basis: Liu and colleagues' 2023 evaluation of four generative search engines distinguished citation recall, the share of verification-worthy statements fully supported by their associated citations, from citation precision, the share of citations that support their associated statements. Citation presence alone does not establish recall, and source support is not independent proof of factual or clinical truth. [6] That historical assessment does not describe every current platform, but it demonstrates why citation presence and evidential support require separate measures.

Human reviewers should use a shared codebook, retain supporting excerpts and assess agreement on independently coded material before resolving differences. AI can assist extraction, but asking the same assistant to validate itself is not independent verification. Report outcomes within defined prompt and platform groups, with answer, claim or citation denominators stated as appropriate. Repeated outputs are not independent human respondents. Preserve refusals and technical failures, and annotate source or platform changes when comparing waves. The result should explain a pattern in tested guidance, not imply a census of what people see.

Studying the answer is not the same as studying its influence

Two experiments answer different questions. Controlled prompting asks whether an input intervention changes the answer. A human experiment asks whether exposure to a defined form of guidance changes understanding, perceived options or choice. The first cannot establish the second. Even consistently infeasible recommendations do not show that users encounter, trust or act on them.

For a proposed shopping experiment, researchers could randomly assign comparable participants to verified guidance conditions covering the same task. If the question concerns presentation, hold the factual information and available options constant while varying how they are presented. If the question concerns information content, define that difference explicitly. Comparing access to a live assistant with conventional search tests a broader experience, including interaction and retrieval, rather than the wording of one answer. Measure understanding of price, quantity and waste trade-offs, and use realistic basket choices with the option to reject the suggestions. Helpfulness ratings alone cannot establish better decisions.

Immediate experimental choices remain different from later purchases. Consented follow-up or linked shopping records can investigate subsequent behaviour, with clear coverage and privacy safeguards. Studies involving health context also require care over sensitive information and participant safety. They should not expose people to harmful misinformation or become personalised medical advice. Properly bounded, these designs connect the guidance available with the consequences of exposure to it, rather than assume that confident recommendations create demand.

Combine the methods to investigate a mismatch

Consider a hypothetical investigation in which surveys and diaries identify affordability, smaller portions and waste as priorities, while repeated AI answers emphasise premium GLP-1-labelled support products. That is a scenario to investigate, not an observed finding. Price records could test whether the suggested purchases fit the budget; qualified evidence review could assess treatment-related claims. Interviews could examine whether shoppers consider those products relevant, and transactions could show actual substitution or repurchase.

The combined evidence would help distinguish explanations that a visibility metric cannot. There might be a genuine unmet need, an unsupported product claim or useful options that the answer overlooks. A retailer could test a different assortment or clearer pack-size comparisons. A publisher or provider might need to correct source information. Human experiments and subsequent purchases could evaluate the response. The purpose is to locate the problem before deciding whether more information, different products or better access would help.

The same household question could also inform separate research into health innovation, affordability and changing retail demand. Treatment adoption, spending and AI recommendations still need distinct evidence, even when they concern the same household.

Why this belongs in MR and Insights

Prompt construction, sampling, experimental contrasts and interpretation are research responsibilities. MR and Insights should lead or co-lead this work with social scientists, technical specialists and relevant domain experts. Marketing and AEO teams contribute knowledge of discoverability, source information and execution. The argument is not that one function owns every AI question, but that evidence used for consequential decisions needs research discipline.

This also changes the brief for AI-visibility measurement. Alongside “Does the provider appear?”, researchers can ask “For which circumstances?”, “With what explanation?” and “Is that option usable?” A gain in recommendations alongside poorer feasibility may not represent progress. A mismatch between human priorities and AI guidance can reveal an important research question, while better source information can address only those problems that are genuinely informational.

AEO Updates Takeaway

AI search gives social science an additional intermediary to investigate. Surveys and qualitative research examine people; controlled answer studies examine how systems interpret their questions; human experiments can investigate what exposure to that guidance changes. For AEO practitioners, the opportunity is to move beyond counting inclusion towards understanding framing, evidence and feasible action. The most useful finding may be an overlooked constraint rather than a missing brand. Research becomes stronger when it studies the guidance alongside the circumstances of the person asking.

References

Sources reviewed 6 November 2026. The six cited sources support methodological precedents or documented platform behaviour. The grocery scenarios, budgets, coding plan and human experiments are proposed applications, not completed research. Published-source review confirms faithful extraction, not empirical reproduction of underlying datasets.

[1] Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms

[2] AAPOR: Social Media in Public Opinion Research

[3] FAQ about Google Trends data

[4] Zhuo et al.: ProSA, Assessing and Understanding the Prompt Sensitivity of LLMs

[5] OpenAI: Searching the web with ChatGPT

[6] Evaluating Verifiability in Generative Search Engines