18 Agentic AI and the Thawing of Frozen Data in Social Science
Pratik Sachdeva, University of California, Berkeley Dennis Feehan, University of California, Berkeley
Abstract: Much excitement about the future of quantitative social science surrounds how it can make what we already do faster and more efficient: for example, agentic AI can accelerate the time from idea to initial results1, it promises faster and more comprehensive literature reviews2, it can help develop and check mathematical derivations3, and it can give high-quality feedback on draft manuscripts4. These parts of “future” have arrived first. But once researchers figure out how to work with these new tools, there will also be changes to what quantitative social scientists do. We will examine these changes by focusing on what we call the thawing of “frozen” data: the ability to more easily and quickly extract structured insight from large, unstructured data sources. We argue that the thawing of frozen data raises the value of social science theory: new data force new theories, and existing theoretical frameworks become more useful. We highlight measurement theory as an example where social science theory has much to offer.
AI usage statement: TBD
18.1 Thawing frozen data
Many rich sources of information about social phenomena have been largely ignored by social scientists because they have been too difficult to fit into the analytical toolkit available to us. These unstructured data sources have been “frozen” in formats that are difficult to unlock because they are highly unstructured. Frozen data include most audio and video, historical documents, as well as multi-mode sources. Using traditional methods – such as systematic manual review and hand-coding – analyzing frozen data was prohibitively resource-intensive and limited to small samples.
In the age of “big data” (roughly the 2010s), new research insights could be unlocked by social scientists leveraging data scientific and machine learning techniques. The result led to the emergence of “computational social science” and its variants. And yet, these techniques were not a panacea: they required programming know-how, machine learning expertise, and access to computational resources (e.g., GPUs) – all of which necessitated invested collaborators or enough time and mentorship to receive the necessary training.
With agentic tools, this is not necessary anymore. For example, a social scientist with a large corpus who needed to label the data according to some schema required (1) obtaining training in machine learning methodology; (2) gathering the requisite data; (3) labeling the dataset, often with undergrad RA to obtain gold labels; (4) training and evaluating the model; and (5) deploying the model. Now, it is completely feasible to write up a prompt and have Claude Code or Codex simply… do all of the above. One no longer needs to really be concerned with the details of training an ML model, or worrying about validation loss, etc.; these are just technical matters to be sorted out by the agent. One only needs a well-posed research question.
The above process can be contextualized in the methodology of “LLM-as-judge”, where LLMs increasingly serve as annotators for any task that can be described in natural language. LLM-as-judge is a heat wave for frozen data via sheer scale: with a well-written prompt, one can extract highly structured, meaningful insights from large corpora with a batch query to your preferred provider”s API and on the order of $10 to $100. LLM-as-judge materialized as a way to interrogate and categorize LLM outputs themselves (via the field of evaluations), but these techniques are relevant for scaling both quantitative and qualitative approaches in social science.
18.2 Measurement theory as a bridge
It”s useful to think of agentic AI as a source of both research tools and also research objects. As research tools, agentic AI can become a part of the research process. For example, qualitative methods can incorporate iterative human-LLM loops: the human loops to iterate on theory; the LLM loops to apply the theory at scale to data. Agentic AI can flexibly apply any number of data scientific techniques to frozen data to unlock insights. There are valid criticisms of these approaches – many of which are derived from the social sciences themselves – but it is undeniable that these tools have already and will continue to reshape the empirical practice of the field.
At the same time, social science has much to offer with regard to understanding agentic models as research objects. For example, AI evaluation has emerged as its own science, with new benchmarks appearing daily. Computer scientists largely lead this charge, trying to identify benchmark datasets and tasks that capture meaningful dimensions on which to assess LLM capabilities. And yet, the science of evaluation is really the science of measurement in disguise – this is a social science problem! Measurement theory already has a long history of developing well-tested frameworks to do the exact thing the field of evaluations is trying to do now.
Measurement theory sits at the intersection of LLMs as research tools and LLMs as research objects. We can leverage the principles of measurement theory – developing valid constructs, interrogating reliability and validity, and rigorously operationalizing what we mean before we attempt to measure it – to build evaluations that actually capture the capabilities they claim to assess. This is particularly important in the context of sociotechnical evaluations, where we assess capability and decision making of agentic AI in messy contexts where there may be no ground truth or verifiable outcome. And leveraging LLMs as research tools benefits from a measurement theory approach, to better ensure that the outcomes of any LLM analysis or annotation couples neatly with the social science theory we actually aim to capture.
18.3 A two-way street
The way measurement theory can help understand and improve LLMs as research tools is part of a larger theme: the coming of agentic AI to social science will be a two-way street. Agentic AI is often described as something that happens to social science. As social scientists gain access to frozen data, new questions can be asked and answered, which will be an opportunity to test existing theories and to create new ones. But the thawing of frozen data also shows that there is a two-way street: the heart of social science is its theory, which has much to offer social science itself and the field of AI at large.