3 New, but Normal? Evaluating AI’s Fitness for Purpose in Survey Research
Soubhik Barari
NORC at the University of Chicago
Trent D. Buskirk
Old Dominion University
tbuskirk@odu.edu
Joshua Y. Lerner
NORC at the University of Chicago
Abstract: As survey research grapples with the risks and rewards of AI, we argue that AI should be treated as normal methodology: recognized for its seemingly unusual properties and novel inferential risks, yet evaluated through “normal” fit-for-purpose, context-specific standards that have governed previous revolutions in survey methodology. We first describe how AI differs from conventional methods across areas of computation, cognition, and coordination, emphasizing that researchers should first clarify which role(s) AI is intended to perform for their particular use case. We then introduce four practical components that researchers should also identify to frame fitness for use assessment: context, stakeholders, success criteria, and alternatives. We also discuss some core trade-offs that evaluations of fitness for use may navigate, including accessibility versus transparency and different cost structures. Finally, we provide an illustrative vignette to demonstrate these considerations in action. By applying our proposed roadmap, quantitative social scientists can begin to treat AI not as methodological novelty, but as normal methodology.
AI usage statement: The authors fully collaborated on the chapter outline, contents and examples. An initial draft was created by the authors and was subsequently revised and refined using a combination of ChatGPT 5.5 Pro, Claude Sonnet 4.6 and Claude Opus 4.8 Max. The tables and figures were constructed by the authors with image editing assistance provided by Claude Sonnet 4.6. Original, author sourced content of the chapter was edited by AI; final editing was done by the authors.
Authors listed in alphabetical order to indicate equal contributions.
3.1 Introduction
Artificial intelligence is increasingly being incorporated into the everyday work of social science. Researchers use large language models and related systems to generate hypotheses, summarize literature, classify text, assist with coding, simulate respondents, support data collection, and automate parts of analysis and reporting. These uses span disciplines and methods—political scientists classifying open-ended responses or campaign messages, sociologists processing qualitative interviews, economists extracting structured information from documents, and communication scholars studying media content at scale. Across these settings, AI is being inserted into workflows that connect conceptualization, measurement, data generation, analysis, and inference.
Survey research makes especially concrete both the promise of AI in social science and the methodological discipline its use requires. Because survey research can encompass the full research workflow—from construct definition and questionnaire design, to sampling and respondent interaction, to data processing, adjustment, analysis, and reporting—it provides a useful setting for exploring how to frame AI’s fitness for use. Recent work has begun to document how AI can enter this workflow across the survey lifecycle, including questionnaire development, pretesting, translation, adaptive interviewing, respondent engagement, fieldwork monitoring, open-ended response coding, nonresponse modeling, synthetic data generation, and automated analysis (von der Heyde et al. 2026; Rothschild et al. 2026). We therefore use survey research as the primary setting for exploring how AI changes where, how, and by whom tasks related to computation, cognition, and coordination are carried out across the research process.
Borrowing from Narayanan and Kapoor (2025)’s influential essay on AI as “normal technology,” we argue that AI should be viewed as normal methodology: used, documented, and assessed according to the same fit-for-use standards that apply to other quantitative social science methods, while recognizing that AI can alter the workflow in unusually broad ways. Our claim is not that AI is ordinary, harmless, or merely incremental. It is that AI should be subjected to familiar standards of methodological justification, with attention to the new forms of uncertainty, flexibility, and consequence its use introduces. Fitness-for-use evaluation is the bridge that makes this possible: it allows researchers to evaluate AI as part of a methodological workflow rather than as novelty alone. Figure 3.1 clarifies what we mean, and do not mean, by treating AI as normal methodology.1
This position follows the path of previous methodological shifts in survey research, including internet surveys (Couper 2008), nonprobability sampling (Jerit and Barabas 2023), and big data (Dever et al. 2020). In each case, survey researchers did not accept or reject the innovation in the abstract. They evaluated its fitness for purpose in relation to specific populations, measurements, error structures, costs, and inferential goals. AI should be subjected to the same discipline, but with attention to the ways it differs from prior tools.
The language of fitness for purpose is therefore central. AI should not be evaluated as a monolithic technology with a single methodological status. It should be evaluated as a family of tools embedded in specific workflows, where its fitness depends on the task it performs and the role it serves. An AI model that is useful for brainstorming candidate survey items may not be fit for producing final questionnaire wording. A system that can triage open-ended responses may not be appropriate for high-stakes classification without human review. Synthetic respondents may support exploratory pretesting or hypothesis development but fail as substitutes for observed human responses when estimating population parameters. The relevant question is not whether an AI system can produce an output or action, but whether that output or action is adequate for the purpose it is meant to serve. We use fitness for purpose as the broader methodological principle and fitness for use as its operational expression: whether a particular AI system, used in a particular workflow, is adequate for the task, context, audience, alternatives, and claims it is meant to support.
Our aim is not to prescribe a single set of metrics for evaluating AI or to offer a universal checklist for adoption. Rather, we offer a roadmap for approaching fitness-for-use evaluation when AI becomes part of a survey or social-science workflow. The roadmap has two steps. First, researchers should identify the role that AI is being asked to do within the workflow—whether it involves computation, cognition, coordination, or some combination of the three. Second, researchers should frame the evaluation problem by clarifying the context in which AI is being used, the stakeholders affected by its outputs or actions, the success criteria by which it will be judged, and the alternatives against which it should be compared. Together, these steps help researchers determine what evidence is needed, what standards are appropriate, and what trade-offs must be weighed when assessing fitness for use in a particular application. The next section takes up the first step by using computation, cognition, and coordination as lenses for understanding AI’s role within the workflow.
3.2 Clarifying AI’s Role
The first step toward evaluating AI’s fitness for use is to understand what kind of contribution AI is being asked to make within the research workflow. Treating AI as normal methodology does not mean treating it as just another version of familiar methodological tools. AI differs from prior innovations in at least three ways. First, it has not reached, and may not soon reach, a stable form. Models, interfaces, training strategies, access points, pricing structures, and performance characteristics change rapidly. Second, operationalizing AI encompasses a wide array of choices, including open versus closed models, prompting versus fine-tuning, local deployment versus API access, and parameter settings such as temperature, reasoning effort, or context length. Third, AI can be used across many parts of the research workflow rather than being confined to a single methodological domain. By contrast, prior innovations, such as computer-assisted interviewing, online panels, and automated text coding, expanded what was possible within specific stages of the survey process.
These features do not prevent AI from becoming normal methodology, but they do require a broader framing of fitness for use, similar to what Buskirk and Conrad (2024) suggest for assessing new and emerging data sources. That framing must look beyond output quality alone. Before researchers can decide what evidence, comparisons, or trade-offs matter, they first need to identify what kind of work AI is being asked to perform within the workflow. We use computation, cognition, and coordination as three lenses for doing so: what AI is being asked to calculate, what it is being asked to interpret or judge, and what it is being asked to manage within a research process. These lenses are not a checklist or taxonomy, nor are they mutually exclusive. Rather, they help clarify the kind of fitness question researchers are facing before turning to the context, stakeholders, criteria, alternatives, and trade-offs that shape a particular use.
3.2.1 Computation
In the computation role, AI can perform tasks that survey researchers might otherwise assign to statistical models, computational algorithms, or machine-learning systems, including imputation, clustering, prediction, forecasting, and classification. In this sense, AI extends a longer trajectory in which machine learning algorithms have become increasingly important to survey research. Buskirk and Kirchner (2020), for example, describe the many ways machine-learning models have been used across the survey lifecycle, from design and data collection to coding, processing, and analysis. Some of the most promising current uses of LLMs build directly on this foundation. For example, many LLM applications in survey research catalogued to date involve open-ended coding (von der Heyde et al. 2026). Prior machine-learning applications for open-ended coding helped establish coding schemes, validation practices, and labeled training data that can now be leveraged by LLM-based approaches to improve scale, flexibility, and performance on more complex coding tasks.
At the same time, functional continuity should not be mistaken for methodological equivalence. Regression estimators are typically grounded in known sampling distributions or explicit assumptions about error. Nonparametric methods still carry assumptions about structure, smoothness, or error properties. Conventional machine-learning methods generally optimize specified loss functions and are evaluated through explicit validation strategies. Language models, by contrast, are trained for next-token prediction, and their outputs are stochastic, prompt-sensitive, and often unstable across model versions and deployment contexts (Farquhar et al. 2024; Steyvers et al. 2025). Model confidence scores or verbal expressions of certainty may be useful diagnostics, but they are not sampling-based standard errors or statistical confidence intervals and may be poorly calibrated to underlying uncertainty (Tian et al. 2023; Yang et al. 2024; Xiong et al. 2024).
The computation lens therefore helps researchers see whether AI is extending an already well-defined computational task, substituting for an existing model or workflow, or generating estimates, classifications, or decisions whose error properties remain unclear. LLMs may be especially fit for computational uses when they build on well-defined ML tasks with existing labeled data and validation benchmarks, but their fitness still depends on task-specific evidence rather than assumed equivalence to prior models.
3.2.2 Cognition
In the cognition role, AI occupies methodological territory previously reserved for humans. A sufficiently capable language model can draft survey items, code open-ended responses, simulate focus group participants, conduct interviews, or summarize data, often at a quality that passes casual inspection. No previous technology has collapsed so many human-facing research roles so quickly or simultaneously (Bail 2024). This is part of AI’s methodological appeal: it appears to extend not only what researchers can compute, but also what they can ask machines to interpret, generate, and judge.
Yet AI cognition is not human cognition. Large language models can flatten demographic variation, produce outputs that shift substantially with small changes in prompt framing, and reproduce stereotypes in systematic but non-human ways (Steyvers et al. 2025; Bai et al. 2025). In open-ended response coding, for example, human respondents may express uncertainty, ambivalence, or resistance to categorization, whereas LLMs may produce categorical outputs even when ambiguity is substantively meaningful (Barari et al. 2025). Evidence from synthetic-respondent applications reinforces this concern. Although LLMs may approximate marginal distributions for some well-established opinions, they often fail to reproduce the joint structure of attitudes, including inter-item associations and subgroup variation (Argyle et al. 2023; Bisbee et al. 2024). For novel or rapidly emerging topics, AI-generated responses may also reflect patterns in historical text rather than the contemporary distribution of opinion (Lee et al. 2024).
AI systems may also behave differently when they are being evaluated. Like human subjects who alter their behavior when they know they are being observed, language models may detect evaluation contexts and shift their outputs accordingly, creating an analogue to the demand characteristics long familiar to survey researchers (Orne 1962; Abdelnabi and Salem 2025; Needham et al. 2025). An AI model may recognize benchmark-like prompts, infer an evaluator’s desired answer from the framing of a task, or produce more socially acceptable synthetic responses when told it is simulating a representative respondent (Greenblatt et al. 2024). This tendency matters because some AI uses are not just problem-solving tasks; they are measurement procedures. Asking a model to explain its reasoning may improve performance in some settings, but when the goal is to study how a model maps inputs to outputs, the explanation request can change the response process itself. The intermediate reasoning a model verbalizes is not necessarily a faithful account of the computation that produced its answer; models may omit cues that influenced their responses or generate plausible but incomplete rationales (Lanham et al. 2023; Turpin et al. 2023; Chen et al. 2025). This resembles a familiar problem in human cognition, where people’s explanations for their judgments may be confabulated or shaped by the act of being asked to explain them (Nisbett and Wilson 1977). Fundamentally, then, the question is therefore not simply whether the output appears plausible, but whether the procedure that generated it is stable and appropriate for the construct or judgment being delegated.
For fitness-for-use evaluation, the implication is not that surface plausibility is irrelevant. In low-stakes or one-off uses, an AI output that appears reasonable to an expert user may be sufficient for brainstorming, drafting, or internal review. The concern arises when cognition-like AI performance becomes part of a repeated measurement, coding, or inferential process. In those settings, an output that looks reasonable in a small test may still drift across prompts, model versions, respondent subgroups, topics, or deployment contexts. What initially appears to be valid interpretation may therefore mask instability in the response process or a mismatch between the construct researchers intend to measure and the construct the model is actually approximating. Researchers should therefore ask what kind of interpretive or judgment task is being delegated, whether surface plausibility is sufficient for that use, and what evidence would show that the delegation remains adequate over repeated use. AI-generated survey items can be reviewed by questionnaire-design experts and tested in cognitive interviews. AI coding of open-ended responses can be compared with trained human coders, especially for responses that are uncertain, mixed, or difficult to categorize. Synthetic responses can be evaluated against survey benchmarks, including subgroup patterns and relationships among items.
The cognition lens therefore helps researchers see when AI is not merely producing text, but standing in for human interpretation, judgment, or response processes. The relevant question is not whether AI can produce an output that sounds reasonable, but whether that output remains dependable enough for the inferential purpose it is meant to support.
3.2.3 Coordination
In the coordination role, AI supports the planning, sequencing, and execution of multi-step tasks within a shared research environment. This may include routing work across tools, managing interactions among researchers and respondents, monitoring field activity, generating follow-up actions, or coordinating multiple automated agents. These uses are appealing because they promise to reduce administrative burden and extend research capacity. Yet coordination also introduces a principal-agent problem that neither human labor nor statistical methods have posed in quite the same way. AI agents capable of pursuing delegated tasks with partial autonomy may be difficult to monitor, constrain, or fully audit (Barari et al. 2026).
For survey research, the coordination problem is especially consequential because the validity of survey data depends not only on individual outputs but also on the integrity of the system through which cases are recruited, contacted, interviewed, processed, classified, and analyzed. AI agents may make it easier to contaminate respondent pools with synthetic cases that evade standard quality controls, threatening the validity of human samples (Westwood 2025). Risks may compound further when populations of AI agents interact, develop shared conventions, or reproduce collective biases that are not detectable by examining any single agent in isolation. In such settings, coordination failures can undermine assumptions about independence, authenticity, and process control that underpin survey sampling and data collection (Ashery et al. 2025).
For fitness-for-use evaluation, the implication is that researchers should consider not only whether an AI system can complete a delegated task, but whether it can do so with fidelity to the research process. This requires attention to monitoring, auditability, accountability, authentication, and failure containment. A coordinated AI workflow may be fit for use when its actions are observable, its decision points are documented, its outputs can be checked against human or procedural standards, and its failure modes do not compromise the validity of the larger study.
The coordination lens therefore helps researchers see when AI is not merely producing an output, but managing steps, relationships, or dependencies within the research system. Coordination adds a fidelity question to fitness-for-use framing: will AI carry out delegated steps in ways that remain aligned with the study’s goals, constraints, procedures, and quality standards?
3.3 Framing AI’s Fitness for Use
The previous section identified the kinds of contributions AI may make within a research workflow: computation, cognition, coordination, or some combination of the three. The next step is to frame the evaluation problem. Researchers need to specify the context in which AI is used, the audience or stakeholders who will rely on or be affected by it, the criteria by which success will be judged, and the alternatives against which AI should be compared.
Methodological judgment has long worked this way in quantitative social science. Researchers do not evaluate a method in the abstract. They evaluate it for a task, within a setting, for an audience, relative to alternatives, and with attention to the inferential claims it is meant to support. AI should be treated similarly. The difference is that AI’s flexibility can make the relevant fitness question harder to see. The same system may act like a research assistant in one setting, a measurement instrument in another, a statistical model in a third, and a workflow coordinator in a fourth. A single global judgment about whether the system “works” is therefore not useful. Fitness for use depends on what the AI is being asked to do and how that use is embedded in the larger research process.
Consider two simple cases. Using AI to assist with a literature review for the design of a specific project is closer to asking a knowledgeable colleague or research assistant for a provisional summary than to deploying a measurement instrument. AI is being asked to perform a computational role here: indexing, systematizing, and summarizing information that may be scattered across sources. But it is also being asked to do cognitive work, because it must judge relevance, identify themes, and decide what details are useful for the project at hand. The use occurs early in the research process, before claims are being made from data, and the output is mainly for the researcher or project team. In that setting, success may be judged by whether the summary is useful, plausible, and sufficiently comprehensive to inform design decisions. The AI output is best understood as comparable to a preliminary human scan of the literature or a graduate assistant’s memo, not as a formal benchmarked procedure. In normal methodological practice, we would not expect such a summary to be perfectly replicable across assistants, nor would we require a formal criterion unless the review itself were the main research product.
By contrast, using AI to code open-ended survey responses is a measurement operation. AI is still being asked to perform both computational and cognitive work: it assigns responses to categories, but it also interprets language, resolves ambiguity, and applies judgment about meaning. The difference is that this work now occurs after data collection, in the production of variables that may support substantive estimates or comparisons. The output is no longer simply a provisional aid for the research team; it becomes part of the basis for analysis. In that setting, success can often be evaluated through agreement with trained human coders, classification reliability, and category-level accuracy. The comparison is not to an idealized human oracle, but to a feasible human or supervised-machine coding process with its own cost, error, and reliability profile. AI used in this way should ordinarily be reproducible, demonstrate acceptable reliability relative to a relevant benchmark, and produce category-level distributions accurate enough to support the inferential claims attached to them.
The same AI system might be adequately fit in the first case and inadequately fit in the second. The difference is not the technology alone. It lies in the work AI is being asked to perform, where that work enters the research process, how directly others will rely on its output, and what would otherwise have been done in its place. Thinking about fitness for use therefore requires attention not only to the capabilities of the AI system itself, but also to the circumstances under which it is deployed and the standards of adequacy implied by that use.
This principle generalizes across the survey lifecycle. Once researchers have identified what AI is being asked to contribute—computationally, cognitively, coordinatively, or some combination of the three—they still need to frame the use case before selecting metrics or judging adequacy.
We focus on four practical questions for doing so:
- What is the context in which AI is being used?
- Who are the stakeholders for the AI-assisted output or action?
- Are there reasonably clear success criteria?
- What are realistic alternatives against which AI should be compared?
These questions do not provide a mechanical verdict, nor do they prescribe a universal set of metrics. Rather, they help researchers determine what evidence is needed, what comparisons are fair, and what standards of adequacy are appropriate for a particular use. This framing step also clarifies how our roadmap relates to more detailed approaches for evaluating AI integration. The AAPOR Task Force on Responsible AI Integration in Survey Research, for example, identifies major fitness-for-use quality dimensions, including validity, performance, sensitivity, and reliability, with more specific metrics nested within those dimensions (Rothschild et al. 2026). Our four questions are not intended to replace those dimensions or metrics; rather, they are intended to help researchers decide when and how such dimensions become relevant for the specific AI use being evaluated. Figure 3.2 clarifies the role of each question in fitness-for-use framing and illustrates the kinds of considerations researchers may need to bring into view before we discuss each question in turn.
3.3.1 What Is the Context?
The first question is context: where AI is being inserted, what type of study or workflow it is part of, and what purpose it serves there. This question extends the computation, cognition, and coordination lenses from the previous section. Those lenses help identify the kind of work AI is being asked to perform; context locates that work within a particular study design, workflow stage, and use. The same AI contribution may carry different fitness requirements depending on whether it is used in polling, qualitative interviewing, focus groups, cognitive testing, mixed-mode or multimodal data collection, internal brainstorming, respondent interaction, data processing, estimation, reporting, or workflow management.
Questionnaire development provides a simple illustration. AI-generated survey questions used early in the design process may be evaluated as draft material for expert review. The relevant concern may be whether the tool expands the range of candidate wording, identifies possible ambiguities, or helps researchers think through alternative formulations. The same AI-generated wording used directly in a fielded questionnaire raises a different fitness question. In that context, researchers may need evidence that respondents understand the item as intended, that the wording does not introduce systematic measurement error, and that the item functions appropriately across relevant subgroups. The AI contribution may look similar in both settings—generating candidate text—but the workflow context changes the evidentiary burden.
Voice interviewing illustrates the same point across study types. Using AI as a voice interviewer for large-scale quantitative data collection is different from using it for qualitative in-depth interviews or cognitive pilot testing. Quantitative administration may prioritize standardization, respondent compliance, and consistent delivery across cases, while qualitative inquiry may require probing, redirection, and situated judgment. As Tirumala et al. (2025) suggest, positive respondent experiences in standardized quantitative interviewing do not automatically establish fitness for qualitative interviewing. Context therefore includes not only the stage of the workflow, but also the kind of study being conducted and the purpose AI serves within it.
Context also clarifies when a single AI use draws on more than one kind of contribution. AI coding of open-ended responses may appear computational because it produces classifications, but it also involves cognition because the model interprets meaning and applies judgment. AI interviewing may involve cognition in the generation of probes, coordination in managing turn-taking and respondent interaction, and computation in monitoring data quality. Fitness-for-use framing should therefore begin by locating the AI contribution within the actual workflow rather than treating the application as a generic use of AI.
3.3.2 Who Are the Stakeholders?
The second question relates to stakeholders or the entities who will rely on, review, interact with, supervise, or be affected by the AI-assisted output or action. Stakeholders may be viewed as audiences who receive outputs delivered from a prior stage: indeed, the ultimate audience of survey research is whoever consumes its insights, but the immediate audience of a particular AI output may be quite different. A questionnaire developer may review AI-drafted items before they ever reach respondents. A field manager may decide whether AI-generated interview transcripts are usable. A coding supervisor may evaluate model-generated classifications. A respondent may interact directly with an AI interviewer. A client, policymaker, peer reviewer, or public audience may consume a report shaped by AI-generated summaries or analyses.
These audiences apply different standards. A questionnaire designer may value breadth, creativity, and identification of possible wording problems. A respondent encountering an AI interviewer may value clarity, brevity, trust, and conversational naturalness. A peer reviewer evaluating an AI-assisted coding scheme may expect explicit decision rules, reproducibility, and evidence of reliability. A client reading an AI-assisted briefing may value interpretability, appropriate uncertainty, and relevance to decision-making. Meeting one audience’s standard does not guarantee meeting another’s. Fitness criteria therefore follow partly from identifying who must trust, use, review, supervise, or be affected by the AI-assisted output at that stage of the workflow.
In many AI-assisted workflows, the relevant stakeholder is also the worker expected to collaborate with the system, rather than just linearly receive its outputs. A survey operations staff member may be asked to monitor AI-generated case notes, a coder may be asked to adjudicate model-suggested classifications, or an analyst may be asked to review AI-generated summaries before they are shared with decision-makers. In these cases, fitness for use depends not only on the quality of the AI output, but also on whether the human collaborator has adequate AI preparedness: the training, context, time, authority, and procedural support needed to understand, question, revise, correct, or override the system. A workflow that assumes human oversight but gives workers little ability to detect errors or challenge model outputs may not be adequately fit, even if the AI performs well in a narrow benchmark.
The voice-interviewing example again illustrates why stakeholders matter. For respondents, the relevant standard may be whether the AI interviewer is understandable, respectful, and easy to interact with. For researchers, the relevant standard may be whether the interview produces usable, comparable, and analyzable data. For field managers, the relevant standard may be whether the system can be monitored and whether failures can be detected quickly. Fitness for use depends on which of these stakeholders must be satisfied and how their requirements interact.
3.3.3 Is There a Clear Criterion of Success?
The third question is whether the task has a reasonably clear criterion of success. Some AI uses have objective functions or benchmarks that can be specified in advance. Open-ended response coding can be compared with trained human coders or gold-standard labels. Classification tasks can be evaluated by accuracy, precision, recall, or agreement. AI interviewers can be evaluated using data quality indicators, breakoff rates, completion rates, respondent experience measures, or interviewer effects (Rytting et al. 2023; von der Heyde et al. 2025; Barari et al. 2025). In these cases, fitness-for-use framing can often point toward familiar metrics or specific accuracy measures, even if the interpretation of those metrics still depends on context, audience, and alternatives.
Other uses do not have a single tractable objective function or success metric. Developing a survey item involves judgment about theoretical fit, face validity, respondent interpretability, sensitivity, burden, and compatibility with an existing battery. Writing a briefing from survey results depends on the audience’s prior knowledge, tolerance for uncertainty, and intended use of the findings. In these cases, the absence of a clear objective function does not mean the use cannot be evaluated. It means that evaluation must rely on other forms of evidence and judgment, such as expert review, cognitive interviews, respondent feedback, sensitivity checks, documentation of assumptions, or deliberation about trade-offs.
The difference can be seen within questionnaire development itself. If AI is asked to identify spelling errors, duplicate items, vague or unknown terms, or inconsistent response scales, success may be relatively easy to define. If AI is asked to draft better items for a complex construct, success is harder to reduce to a single metric. Researchers may need to ask whether the item reflects the construct, whether respondents interpret it as intended, whether it fits the surrounding battery, and whether it works across relevant subgroups. The task may still be evaluable, but the evaluation is not reducible to simple accuracy. Buskirk et al. (2025), for example, experiment with using LLMs to generate survey questions and evaluate the resulting items using a combination of qualitative and quantitative criteria drawn from Dillman et al. (2014), rather than relying on a single objective measure. Fuchs et al. (2026) illustrate a different possibility by using the Survey Quality Predictor (SQP) score as an objective function for training an LLM to generate survey items. Yet that example also shows why the objective-function question must be tied to context: the SQP is most directly relevant to attitudinal items, so even when an objective measure can be used, its scope may be restricted to particular types of items or measurement goals.
The AI-as-respondent formulation illustrates the stakes even more sharply. Using AI-generated outputs for power analysis or early question pretesting may require only approximate fidelity to human response behavior if the purpose is to guide study planning. By contrast, using AI-generated responses as substitutes for or supplements to public opinion data requires making inferential claims about human populations based on synthetic evidence. That use carries substantially higher evidentiary demands and has documented risks even when models are conditioned on demographic variables (Bisbee et al. 2024; Lyman et al. 2025). Where no clear objective function realistically exists, fitness for purpose cannot be established by benchmark alone. Researchers must instead clarify what “good” means in the project context and what forms of evidence are sufficient to support the intended use (Argyle et al. 2026; Davidson 2026).
3.3.4 What Are the Alternatives?
The fourth question is: compared to what? When an AI-assisted procedure is judged adequate, it is adequate relative to some alternative. That alternative might be a human expert, a trained coder, a conventional statistical model, a rule-based procedure, the existing production workflow, a lower-cost design, or no procedure at all. Naming the alternative is essential because the fitness judgment often turns less on whether AI is flawless than on whether it is adequate relative to what would realistically be done otherwise.
For cognition-oriented uses, the alternative is often human judgment, but human judgment should not be idealized. Benchmarking AI coding against human-coded labels treats human coding as the reference point, yet human coders exhibit intercoder unreliability, drift, fatigue, and disagreement. The relevant comparison is not to an oracle but the feasible human process, with its actual error rate, cost, and time demands. Holding AI to a standard that the realistic human alternative also cannot meet would make the assessment of fitness more stringent than the practical comparison warrants.
For computation-oriented uses, the alternative may be a purpose-built statistical or machine-learning method. An LLM asked to impute missing values competes with multiple imputation. An AI model asked to predict or classify competes with supervised methods such as penalized regression, random forests, or gradient boosting. An LLM asked to forecast competes with established time-series methods. The comparison is not only whether AI performs well on point accuracy, but also whether it provides the inferential properties needed for the downstream claim. Multiple imputation, for instance, exists not only to fill in missing values but to propagate imputation uncertainty into final estimates (Rubin 1987). An LLM that imputes plausibly but provides no usable characterization of uncertainty may be poorly fit for inference that depends on that uncertainty, even if its point predictions look strong.
Readability assessment offers a more bounded computational example. Olson and Buskirk (2026) report that many LLMs used to compute readability statistics for survey items still lagged behind standard tools such as readable.com or implementations in R. In that setting, the relevant comparison is not a human expert but an established computational procedure with clearer rules, greater reproducibility, and stronger accuracy for the specific task. Relative to those alternatives, the LLMs were not sufficiently accurate to warrant their use for in situ readability assessment, even if they may be useful for adjacent tasks such as explaining difficult wording or proposing revisions.
For coordination-oriented uses, the alternative may be a human-managed workflow, a rule-based automation, or a supervised hybrid system. The comparison should include not only speed and cost but also process fidelity, transparency, auditability, and failure containment. An AI agent that coordinates respondent follow-up more quickly than a human manager may still be unfit if its actions cannot be monitored, if it deviates from protocol, or if failures contaminate the larger study process.
The alternatives question also helps explain why the same AI use may be fit in one setting and unfit in another. An AI-generated literature summary may be adequate when the alternative is a quick preliminary scan or no review at all, but inadequate when it is expected to replace a systematic literature review. Similarly, synthetic respondents may be poorly fit as substitutes for real public-opinion data, where the alternative is observed human responses with known properties. The same synthetic responses may still be useful for power analysis or early pretesting, where the alternative may be rough assumptions or no empirical guidance (Bisbee et al. 2024; Lyman et al. 2025). In each case, the fitness judgment changes because the realistic alternative changes.
Survey methodology has navigated this comparative logic before. Evaluations of nonprobability samples often turn on whether the sample is adequate relative to a probability benchmark for a particular inferential goal, or relative to the reality that a probability sample may be infeasible (Jerit and Barabas 2023). Evaluations of lower-cost instruments similarly weigh a cheaper design against the standard it is meant to approximate, rather than against perfection (Dever et al. 2020). The point is not that AI should be held to a lower standard, but that fitness for use can involve both absolute and comparative judgments. Researchers must still ask whether AI is adequate for the task, context, audience, and claim at hand, but that judgment should be made in relation to the realistic alternatives available.
3.4 Assessing AI’s Fitness for Use Requires Weighing Trade-offs
Fitness for use is rarely binary. AI may improve one part of a research workflow while creating new risks elsewhere: speed may come at the expense of transparency, consistency may come at the expense of construct validity, and lower marginal costs may require greater monitoring and oversight. The trade-offs also differ depending on whether AI is being used primarily for computation, cognition, coordination, or some combination of the three. A computational use may raise questions about accuracy, uncertainty, and inferential tractability. A cognitive use may raise questions about interpretive quality, respondent experience, and construct validity. A coordination use may raise questions about delegation, oversight, process fidelity, and accountability. In practice, many AI applications involve more than one of these contributions, which is why fitness arguments must often be holistic rather than tied to a single metric.
3.4.1 Error and Quality Trade-offs
AI’s effects on survey quality are not uniform. Improvements along one dimension of Total Survey Error can come with losses along another (Groves 1989). AI-assisted conversational interviewing, for example, may reduce some interviewer effects by standardizing question delivery, probe wording, tone, or pacing. It may also reduce costs or make interviewing more scalable. At the same time, it may introduce new measurement risks if its conversational logic departs from standardized protocols, if respondents react differently to AI-mediated interaction, or if the system adapts in ways that undermine comparability across cases. It may also create coverage or nonresponse concerns if some respondents are unwilling or unable to engage with AI-mediated communication. In this case, AI may improve coordination and standardization while creating new risks for measurement, coverage, or response quality.
AI-assisted coding presents a different quality trade-off. A model may improve throughput and consistency relative to a team of human coders, especially for large volumes of open-ended responses. But if it systematically misclassifies responses from particular demographic groups, subcultures, dialect communities, or topic domains, efficiency gains may come at the cost of construct validity or subgroup comparability. The fitness argument therefore cannot be “AI is faster” or “AI agrees with coders on average.” Researchers must ask whether gains in efficiency, consistency, or scale outweigh the risk of introducing new errors that affect construct validity, subgroup comparability, or the intended estimates and claims.
3.4.2 Access, Transparency, and Replicability Trade-offs
A second set of trade-offs concerns access, transparency, and replicability. Proprietary, closed-source models are often among the most capable and easiest to deploy. They may require little specialized infrastructure and may allow researchers, practitioners, or smaller teams to experiment with AI-assisted workflows quickly. But they are also opaque: researchers cannot inspect model weights, audit training data, or fully characterize how errors may change across prompts, model versions, or deployment settings (Barrie et al. 2025).
Open-source or locally deployed models offer greater transparency and replicability in principle, but they may impose higher burdens in compute, storage, engineering time, security review, and maintenance. Bail (2024) emphasizes that broad access to powerful AI tools can expand the scope of social science research while also weakening the transparency and replicability that give findings their credibility. The public and many practitioners may favor accessibility and usability, while cumulative scientific knowledge often depends on auditability and documentation (Brodeur and Barbarioli 2026).
Choosing between closed and open-source models is therefore not a purely technical decision. It encodes a judgment about whose requirements take precedence: the practitioner who values usability and immediate access, the reviewer who values auditability, the respondent or data subject whose privacy may be implicated, or the scientific community that depends on replicability. This trade-off is also not symmetric. Institutional resources, technical capacity, and existing infrastructure shape which option is actually accessible to a given team, with consequences for who benefits from AI-enabled efficiencies and who does not (Eads and Dokshin 2026).
3.4.3 Cost, Speed, and Oversight Trade-offs
AI also changes the cost structure of research workflows, which is one reason fitness for use cannot be assessed by output quality alone. Although many AI tools appear inexpensive or free to access, applied use may involve token budgets, API charges, compute costs, hosting, privacy controls, monitoring, and workflow management. Closed-source models often operate on a per-use pricing model, with variable costs that scale with tokens, calls, or volume and require little upfront infrastructure investment. Open-source or locally deployed models may involve fixed or sunk costs, including compute hardware, storage, engineering time, deployment, monitoring, and maintenance. For a small research team running a single project, a pay-per-token API may be cheaper and simpler. For a large organization with sustained, high-volume AI use, local deployment may be more economical despite the upfront investment. Neither option dominates universally. The better choice depends on scale, technical capacity, privacy requirements, and tolerance for variable versus fixed costs, conditions that vary across institutions and can shape who benefits from AI-enabled efficiencies (Eads and Dokshin 2026).
Cost and speed gains also create oversight trade-offs. AI-generated summaries, classifications, recommendations, or case notes may increase throughput, but they may also create pressure for workers to accept outputs quickly rather than examine them carefully. If an AI-assisted workflow depends on human review, then reviewers need enough time, authority, and support to perform that role meaningfully. A workflow that claims to keep humans “in the loop” may still be poorly fit if workers lack the training, context, or authority to question, correct, or override model outputs.
This issue is especially important for coordination-oriented uses. AI may route cases, schedule follow-ups, generate interviewer prompts, summarize field activity, flag anomalies, or coordinate multiple tools. Cuevas et al. (2025) is a proof-of-concept study of AI agents for conversational interviewing that illustrates both the appeal and the challenge of this kind of delegation: automating research tasks may reduce burden and expand capacity, but it can also introduce practical quality concerns, such as whether the system acts at an appropriate pace and whether its decisions remain observable and correctable. The more autonomy AI is given, the more researchers must ask whether its actions are timely, auditable, and aligned with the study protocol. Delegation may reduce burden, but only if monitoring and accountability remain proportionate to the risks of the task.
3.4.4 Standardization and Adaptability Trade-offs
AI can support standardization by applying the same rules, prompts, or procedures across many cases. It can also support adaptability by tailoring probes, explanations, examples, or follow-up actions to a particular respondent or case. Both capabilities can be valuable, but they can pull in different directions. Standardization supports comparability, replicability, and process control. Adaptability may improve engagement, accessibility, and responsiveness to complex or unexpected situations.
Conversational interviewing illustrates the tension. In standardized survey administration, adapting too much may threaten measurement equivalence across respondents. In qualitative interviewing or cognitive testing, adapting too little may undermine the inquiry process and reduce the richness, depth, or diagnostic value of the information collected. Similarly, AI-generated explanations or clarifications may help respondents understand a question, but they may also change the stimulus across cases. Whether adaptability is a strength or a liability depends on the context and on the inferential claim the data are meant to support.
3.5 Bringing Fitness for Use into Focus: From Roadmap to Assessment
The roadmap we propose in this chapter is intended to help researchers move from a general question—whether AI “works”—to a more structured account of what role AI is being asked to perform, where it enters the workflow, who depends on it, what would count as success, and what alternatives are available. Figure 3.3 showcases an applied vignette to illustrate how this reasoning can be used to bring the relevant fitness-for-use considerations into focus before selecting specific quality dimensions, metrics, or evidence for evaluation.
The vignette scenario highlights the central trade-off. The same features that make the LLM attractive—speed, scale, and consistency—may undermine qualities that matter for the task, such as preserving nuance, handling ambiguity, or maintaining valid comparisons across groups.
As a result, the system may be fit for generating preliminary themes, triaging responses, or drafting summaries, yet unfit as a fully automated substitute for human coding when outputs support substantive conclusions or policy decisions as official statistics. The roadmap helps surface these competing considerations, but it does not determine the answer. Instead, it points researchers toward the quality dimensions that should be examined—such as reliability, validity, subgroup performance, transparency, and oversight—and toward the evaluation evidence needed to assess them. In the AI-assisted coding example, that evidence might include coder agreement and category-level accuracy for reliability and validity, subgroup comparisons for differential performance, review of ambiguous cases for construct interpretation, and assessment of whether the amount of human review required is compatible with the claimed efficiency and quality gains.
3.6 Conclusion
No matter the area of application within the survey or social-science workflow, the task is not to decide whether AI works or whether it is trustworthy in general, but to determine when, for whom, compared with what, and at what cost it is fit for use. Survey methodology has long absorbed new technologies by asking this kind of question: not whether internet surveys, nonprobability samples, or computational text analysis were universally valid, but where, how, and for what purposes they were good enough. AI belongs in that history, even as it stretches it by entering the roles of computation, cognition, and coordination at once. Only when judgments about AI become explicit, holistic, contextual, and evidence-based can AI move from methodological novelty toward normal methodology.
By methodology, we mean the set of procedures, assumptions, evaluative standards, and reporting practices through which researchers produce and justify evidence. As Figure 3.1 clarifies, AI is not a methodology in every use: it is a general purpose technology that appears in research as a family of tools that can be integrated into methodology. AI becomes methodologically consequential when it changes how constructs are operationalized, data are generated or processed, analyses are conducted, workflows are coordinated, or inferential claims are justified within existing methodologies. This distinction lets us avoid treating AI as either magic, merely software, or even a singular methodology meant to replace existing ones like qualitative methodology or CATI interviewing.↩︎