8  Failing Faster, Learning Better: Agentic AI and Empirical Social Science Research

Charles Crabtree
School of Social Sciences, Monash University, and K-Club Professor, University College, Korea University

Valentina González-Rostani
Department of Political Science and International Relations, University of Southern California

Jae Yeon Kim
Department of Public Policy, University of North Carolina at Chapel Hill

Abstract: How can social scientists use agentic AI to produce not more, but better work? We argue that the answer lies in using agentic AI to accelerate iterative search by generating alternatives, testing them, and failing more productively. Drawing on Jonathan Bendor’s framework of innovation as a search problem that balances creativity (broad exploration) and criticism (rigorous evaluation), we organize our chapter around two recurring search failures: creativity failure, in which promising ideas are screened out before they are adequately explored, and criticism failure, in which researchers invest too much in weak research designs. Within that framework, we then discuss two applications. One focuses on organizational-level measurement research, where agentic AI can lower the feasibility barrier to exploring new projects and automate iterative validation, thereby avoiding creativity failure and catching flawed measures before they enter the downstream workflow. Another focus is on experimental research, where the prevalent error might be criticism failure arising from underpowered, under-piloted designs. In this domain, agentic AI enables stress-testing of designs and the implementation of construct validity checks at scale. Though AI agents can be potentially transformative for empirical social science research, we also see key limitations, such as inferences from silicon samples not generalizing to real humans; LLMs defaulting to agreement; AI-generated stimuli carrying biased and multi-barreled treatments; and new problems created by using AI to defend against AI contamination. Agentic AI can be used to improve design, measurement, and validation, but it cannot replace human judgment.

AI usage statement: The authors used Warp.dev and Claude Opus models to generate the detailed outline for the chapter. The prompt used to generate the detailed outline included the authors’ content and a preliminary outline. All authors substantially rewrote the detailed outline. They also used Claude Opus and Codex GPT 5.4 models to assist with the literature review and to clean up the citations/bibliography. The authors checked the citations manually and used Grammarly to correct spelling and grammatical errors.

All authors contributed equally. The names are listed alphabetically by last name.

“Every genuine test of a theory is an attempt to falsify it, or to refute it. Testability is falsifiability; but there are degrees of testability: some theories are more testable, more exposed to refutation, than others; they take, as it were, greater risks.”
Conjectures and Refutations, Popper (1963, 36)

“And policy analysts, in deciding whether to use a finding from behavioral decision theory, are in the FDA’s position: they can be either too patient, and fail to apply an idea that is ready, or too impatient, and apply something prematurely.”
Bounded Rationality and Politics, Bendor (2010, 182)

8.2 Innovation as Iterative Search: The Bendor Framework

In Karl Popper’s framework, scientific progress depends on subjecting theories to falsification. The more testable a theory is, the better (Popper 1963). Jonathan Bendor (1950–2025), the late professor of political economy at Stanford’s Graduate School of Business, extends Popper’s framework of scientific discovery to organizational problem solving (Bendor 1995, 2010, 2015). Building on Simon (1947)’s work on bounded rationality and Lindblom (1959)’s work on disjointed incrementalism, he argues that innovation arises from balancing two risks: dismissing promising ideas before adequate exploration (creativity failure) and overinvesting in weak ideas (criticism failure). Because these risks trade off, effective research requires both creativity and criticism.

Agentic AI can compress creativity and criticism into a streamlined process. It reduces the cost of local iteration. On the creativity side, agents can lower the marginal cost of trying new measurement strategies and testing alternative designs, helping researchers possibly avoid the creativity failure of dismissing promising ideas before they are adequately explored. On the criticism side, they can automate checks against benchmarks, across specifications, and under adversarial prompting, which helps guard against the criticism failure of overinvesting in bad or weak ideas.

This epistemological approach, or innovation workflow, creates structured redundancy. Redundancy is often seen as waste, but it can improve reliability in high-risk systems (Landau 1969). Multiple identity checks at an airport and the different, overlapping health protocols during COVID-19 illustrate this logic. The point of these overlapping checks is to reduce the chance that a failure goes undetected.

Table 8.1 illustrates how creativity and criticism failures map onto our two applications, organizational measurement and experimental research, and serves as a roadmap for the sections that follow.

Table 8.1: Creativity and Criticism Failures across Applications
Application Creativity failure (missed opportunities) Criticism failure (overinvestment in weak ideas)
Organizational measurement Feasibility constraints screen out data-intensive projects. Early commitment to flawed measures that propagate downstream.
Experiments and audits Underpowered or weak designs fail to provide evidence of underlying treatment effects. Contaminated or weakly validated evidence appears credible.
Role of agentic AI Expands research by lowering feasibility and iteration costs. Improves evaluation through iterative validation and stress testing.

8.3 Application 1: Organizational-Level Data and Measurement

8.3.1 The Problem

Research on organizations routinely confronts severe data constraints. Relevant evidence often sits in semi-structured sources outside standard datasets, including speeches and dialogues by local actors (e.g., politicians, citizens, and interest groups), oral histories, administrative records, and organizational websites (Grimmer and Stewart 2013; King et al. 2017; Parthasarathy et al. 2019; Barari and Simko 2023; Kim et al. 2025). These sources are often essential for understanding what organizations offer their members, how they build networks and coalitions, how they develop and deliberate over agendas, and how they shape political and policy outcomes. The challenge is not only in collecting such data, which can be sparse and costly, but also in ensuring measurement validity: whether the observations researchers assemble actually operationalize the concept of interest in a valid way (Adcock and Collier 2001; Grimmer et al. 2022).

Through Bendor’s lens, these constraints generate two kinds of search failure. The first is feasibility-filtered omission, which is a creativity failure in Bendor’s sense. This failure occurs when research programs are not pursued because the required data are too costly to assemble. The second is premature commitment, which can be thought of as a criticism failure. When data annotation is done by hand, researchers might commit too early to a single operationalization. If that measure proves flawed, reannotation is prohibitively expensive. For instance, a team studying a political party’s immigration rhetoric might discover after six months that their scheme conflates economic and cultural frames. By that point, the flawed measure has already propagated into downstream work, including data analysis and visualization.

8.4 Application 2: Experimental and Audit-Based Research

8.4.1 The Problem

Experiments are often lauded as the “gold standard” of empirical research because they can provide evidence about the causal effect of treatments on outcomes, or, more generally, about how one thing affects another. These designs are important not only for helping us understand how the world broadly works, but also because policies are often based on that understanding. For example, survey experiments can help identify the effect of various information interventions on individual attitudes, beliefs, and stated actions (Druckman and Green 2021). Audit experiments, for example, can identify how group differences influence individual behavior (Block et al. 2021; Gaddis et al. 2022; Butler and Crabtree 2017). While these and other experimental design types are very powerful research tools, they require both tremendous resources and careful forethought to implement well.

One particular concern that experimentalists must grapple with is the possibility that they will spend countless hours on an experiment, implement it, find no effect of their intervention, and only then realize they did something wrong. Kane (2025) identifies seven pathways to this unfortunate outcome. These include respondent inattentiveness, manipulation failure, pre-treatment threats, poor outcome measurement, ceiling and/or floor effects, countervailing effects, and small samples. This last issue might be both particularly important and generally underappreciated. Arel-Bundock et al. (2026) show that quantitative political science research is “greatly underpowered,” and often conducted on samples so small that they cannot be used to reliably provide evidence of hypothesized effects. While small samples can lead to null results, they can also, more problematically, yield estimates that are unstable in sign and inflated in magnitude (Gelman and Carlin 2014).

In our framework, this set of issues can be viewed as various instances of a criticism failure. Researchers often invest substantial time and effort in designs that are underpowered, underpiloted, or otherwise too weak to generate reliable evidence. This risk of insufficient validation is even more serious now. Online panels, once central to the globalization of public opinion research (Heath et al. 2005; Thomas 2024), may now be contaminated (to some unknown degree) by AI agents. Early evidence suggests that the scale of the problem is already substantial. About one-third of crowdworkers report using LLMs to answer open-ended survey questions (Zhang et al. 2025). Recent evidence also suggests that, in live surveys, responses that are LLM-mediated, AI-assisted, or fully AI-generated likely account for between 4% and 45% of all responses (Westwood and Frederick 2026; Panizza et al. 2026; Rilla et al. 2025), and that AI agents are able to autonomously complete surveys and pass standard response-quality checks (González-Rostani and Raviv 2026).2

8.5 How Agentic AI Improves Experimental Research

AI agents can help improve instrument design and thereby maximize the probability of minimizing what Bendor would consider a criticism failure, through pretesting. Researchers can use synthetic response data to help diagnose confusing items, improve the construct validity of treatments, and detect possible order effects. Agents can also produce large sets of ecologically valid treatment materials, such as cover letters, emails, and profiles, built on real-world materials and calibrated to specific contexts, expanding the range of feasible audit studies or experimental vignettes. In addition, agents can help construct LLM-based treatments that deliver personalized, persuasive messages and conduct multi-turn conversations that adapt to participants’ responses (Argyle et al. 2025; Costello et al. 2024; Crabtree et al. 2025) or develop survey interfaces that emulate real-world behavioral contexts. In these ways, AI agents then help expand the feasible experimental design space by helping improve construct validity and ecological validity, and by lowering the cost of creating interactive and personalized treatments.

Agents also have clear limits, though. One of them concerns synthetic data, where agents often cannot be trusted to accurately simulate human responses (Bisbee et al. 2024). This is because LLM simulations reproduce average responses from their training data and tend to produce data with lower variance, as well as potentially less valid responses for new question items and response options. Another limitation is regarding the ability of AI agents, and LLMs more generally, to detect ‘fake’ responses from other agents. Adversarial agents can stress-test survey instruments, redesign quality checks, and benchmark suspicious responses (González-Rostani and Raviv 2026), but the threshold for what constitutes a suspicious response is a moving target (Westwood 2025). The core issue is not only technical but also substantive and institutional: data authenticity and reporting quality. Researchers must assess the credibility of responses and be transparent about detection procedures. AI-assisted contamination detection is, therefore, a short-term fix, not a long-term solution.

8.6 Conclusion: What These Tools Can and Cannot Do

Agentic AI offers genuine researchers the promise of tremendous value by compressing the exploratory and evaluative loops through which empirical social science advances. Agents reduce creativity failures by expanding what can be tested and reduce criticism failures through benchmarking and adversarial stress-testing. The benefits of AI agents are clearest where feasibility constraints have historically filtered out research programs, as in organizational measurement, and where underpowering or high resource demands are the pressing concerns, as in experimental design.

But this alluring promise comes with clear limits at the moment (and possibly in the far-off future as well). First, LLMs suffer from acquiescence bias, defaulting to agreement. The criticism benefits we outlined above only work if agents are instructed to find failures. Without that adversarial structure, cheaper exploration can simply generate more criticism failures by producing polished but weak outputs. One way to avoid this issue is to have an agent from a different model engage in criticism. Second, silicon samples, despite their promise, are not the same as pilots with actual respondents. They cannot be used to reliably establish treatment effects in real populations (Xie et al. 2026). Using these samples as if they might cause researchers to systematically favor designs that match the AI’s priors. Third, AI-generated stimuli can carry forward hidden biases and even potentially perpetuate them. These can undermine construct validity and create ethical concerns and potentially subject harms. Fourth, faster pipelines do not, per se, guarantee better work: without techniques of adversarial validation, prompt preservation, and model versioning, and comprehensive human review, researchers may simply produce opaque or weakly grounded designs more quickly. In addition, models change, so pipelines must be versioned because results may not replicate across configurations (Bisbee et al. 2024).

We see the biggest danger in this new agentic world in the possibility that researchers will become over-reliant on AI at the expense of cultivating human judgment and discretion. Social science could become a field in which AI generates treatments, tests them on synthetic respondents, analyzes the results, writes them up, and submits them to a journal, where they are then reviewed by another AI agent. We do not have to follow this path, though, and anything akin to automatic research workflows should be avoided in our view. Freedman warned that “there is a strong desire to substitute intellectual capital for labor” (Freedman 1999, 255). The temptation is to take shortcuts. In an earlier period, this often took the form of researchers relying on model-based inference, such as regression, without critically addressing its underlying assumptions. Agentic AI is more flexible and, therefore, more widely applicable. But usable does not necessarily or always mean useful. We should resist the temptation to mistake convenience for validity - or sacrifice the latter for the former. Academic research requires more than efficiency. Credibility depends on verification. Design decisions, theoretical interpretations, and empirical validation remain human responsibilities.

  1. It also allows researchers to take their original documents and quickly translate them for other audiences, such as an interactive dashboard of public comments on a regulatory issue.↩︎

  2. At the same time, the institutional basis for iteration has become more fragile. Federal support for social science research is under pressure, including proposals to eliminate the National Science Foundation’s Social, Behavioral, and Economic Sciences Directorate, the major federal funder of academic social science in the United States (Kozlov et al. 2026; American Political Science Association 2026).↩︎