12 AI and the Measurement Imperative
Rehan Mirza
Annenberg School for Communication, University of Pennsylvania
Sandra González-Bailón
University of Pennsylvania
sandra.gonzalez.bailon@asc.upenn.edu
Abstract: The social sciences emerged from the sweeping transformations triggered by the Industrial Revolution. The quantification of social life became a tool to understand societal change and propose evidence-based reforms. Here we argue that in the AI era measurement is even more crucial to the social science mission, not just to document transformations and guide social reform, but also to define human benchmarks and boundary markers for domains that should not be reduced to quantities. The choice of which measures should guide the research workflow must remain fundamentally human, or we risk losing control over the priorities that shape the pursuit of knowledge for the benefit of all.
AI usage statement: AI was not used.
12.1 Introduction
It has become commonplace to compare the rise of AI with the Industrial Revolution. The latter took a hundred years to fully unleash while the former is taking shape in the scale of a decade. But there is one key aspect that both eras have in common: the depth of the social transformations triggered by technological breakthroughs. The spinning and steam engines sent ripples through every layer of society, affecting how we work, live, communicate, and organize; so is deep learning and the many models it helps create: there is no domain of human activity that will remain untouched by AI technologies. The two eras share another similarity: the role the social sciences play in making sense of these transformations. Beatrice Webb, a sociologist studying the consequences of the Industrial Revolution (and one of the founders of the London School of Economics in 1895), made an echoing point: you cannot improve what you cannot measure.
Today, the measurement imperative is equally central to the social sciences mission. It is a requirement if we are to assess the social impact of AI with the hopes of stirring change. Webb and her contemporaries campaigned for evidence-based reform to improve the living conditions of the working class, whose lives had been radically transformed by the automation of labor (Webb 1926). Social research must once again establish the key facts about how AI will reshape labor markets, institutions, and human behavior if we are to keep pace with the disruption and regulate its deployment based on data. But if numerical measurement is needed for credible research (to demonstrate impact, quality, truth, or progress), in this AI era we also need measurement for another, more fundamental, reason: to define human benchmarks.
Measurement plays a dual role: it enables credible research, which we can conduct using AI, and it helps us define the human capabilities against which AI is evaluated (both in research and other domains). The definition of benchmarks helps us decide what to automate and what to preserve for human mastery. These benchmarks are implicit in our definitions of what counts as intelligence, what stands as good judgment, or what it means to be connected to one another when machines increasingly do the thinking and judging. We cannot determine if machines operate as intended without performance measures. As of today, the most consequential measures of performance are in the hands of the companies developing the AI models. These companies define their success using metrics they select for themselves.
12.2 The Double Edge of Measurement
In science, it is often said that you do not understand what you cannot measure. In the social sciences, we recognize that many human concepts (e.g., creativity, consciousness, social cohesion, moral judgment) resist quantification. And yet we aspire to find better proxies, if only to delimit the space of what we consider uniquely human territory. We also know that measurement often changes behavior: when a measure is used as an indicator for decision-making, it soon ceases to be a good measure (Campbell 1979). The reason is that incentives realign around the metric itself, rather than the original goal. Social media optimizing for engagement is a prominent example of how distorting this phenomenon can be. Engagement was originally conceived as a proxy to content quality, under the assumption that valuable material captures and holds attention. However, as the metric also served commercial interests, it ended up measuring time spent on platforms rather than actual content quality. In fact, the assumed association was soon reversed: low-value material, the kind that plays to our baser instincts, proved far more effective at gripping attention, ultimately crowding out high-quality content.
There are other examples closer to home: our measures of academic productivity will soon lose their meaning, as papers that once took months to produce (and much human mastery) can now be generated in a matter of days with the help of automated pipelines. We forget that productivity was never the point. The point was the learning and the craft and the development of shared structures of understanding. Systems that optimize for productivity risk the same fate as social media platforms that optimize for engagement: the metric improves while the underlying quality collapses. This is evidenced by the spike in hallucinated references reported in recent scientific research (Topaz et al. 2026; Zhao et al. 2026). Rewarding the mere appearance of activity creates a dangerous incentive structure where accuracy becomes secondary to output.
The social sciences have always paid special attention to the institutional arrangements that sustain knowledge creation. AI is reconfiguring those arrangements in ways that force us to rethink what we measure and with what purpose. The question of whose knowledge is treated as authoritative will be key, especially given the opacity that surrounds the development of AI models. Measurement serves a dual purpose: it brings transparency to decision-making by making choices visible, comparable, and contestable; but it also serves those who want to constraint and control choices. Early social scientists embraced the first purpose to provide accountability during a period of deep social disruption. Today, social scientists must reclaim data and metrics from private interests that control and exploit them for their own benefit.
12.3 AI and the Research Workflow
It is already common to insert AI tools into the different stages of the research workflow to augment scientists’ capabilities. Our argument in this position piece is that deciding what to measure remains the most important stage of the research process. It has always been key, but in the AI era it is decisive: losing control over what we measure means losing control over our knowledge priorities. The good news is that we can now unleash our research imagination with less anxious attention to resource constraints. Feasibility always imposes a ceiling on what can be achieved, but now the proverbial genie-in-a-bottle is out, ready to grant our wishes. Thanks to AI, non-traditional sources of information can now be treated as data. For example, recent research uses video footage of urban spaces and relational data buried in text to capture, with the help of AI, relevant social facts.
The first example derives from a recent paper that makes use of computer vision and deep learning to analyze video footage of public spaces in three US cities during the late 1970s and the late 2000s (Salazar-Miranda et al. 2025). One of the high-level findings of this research is that city streets are acting as corridors for movement rather than as spaces for social interaction, which has implications for how we design public spaces and how we can foster social connections and community-building. The second example comes from another recent paper that uses LLMs to extract social network data from the acknowledgement section of hundreds of thousands of published articles (Danús et al. 2026). These informal ties create structures of support and a type of social capital that influences the process of knowledge generation and dissemination. Science, after all, is a collective endeavor and mapping how scientists self-organize can help foster more effective forms of collaboration.
AI tools augment our ability to measure intangible social facts like the use of public spaces or the social-support structures we depend on – facts we could not easily quantify before these tools were available. Early social science struggled due to “the sheer difficulty of obtaining numerical information on social topics” (Lazarsfeld 1961). Even as quantitative methods grew more sophisticated, it remained true that not everything that could be counted did actually count much (Cameron 1963). Because so few things of interest were measurable at scale, quantitative social science overused the same metrics, exploiting known data sources and consequently engaging in too little exploration. Now, AI tools allow us to push into uncharted territory, expanding both what counts and what can be counted. These new capabilities will be essential as social research studies the very transformations AI is producing. Establishing rigorous benchmarks is critical to objectively measure these capabilities and their societal impact.
12.4 Benchmarks in AI Research
When AI is used not to uncover structural facts but to infer aspects of human skill or behavior, benchmarking becomes necessary. Benchmarks combine data and metrics to define tasks and capabilities: they offer a shared reference point to compare different models. In the case of AI, the comparison of interest is algorithmic performance vis-a-vis human behavior. Benchmarks allow us to track progress and reveal errors. But to judge whether AI performs at human levels, we first need to establish where those levels actually lie. The question here is not just where to find the right data source and how to best extract meaningful measures from new sources. The key element is how we define the tasks and capabilities of interest. A research priority for the social sciences in the AI era is to develop a framework that clearly specifies which dimensions are valid to measure in which contexts.
In developing this framework, a familiar reminder applies: the map is never the territory. The metrics we use to define tasks and capabilities are always abstractions, partial representations of the far richer human experience. Abstraction is a fundamental part of the research process, but it works only if social scientists retain the authority to decide which representations are appropriate for a given domain. This also includes deciding which aspects of human life should resist simplification, such as the inherent messiness of democratic practice. Again, not everything that counts can be counted.
There are at least two types of benchmarks social science will need to help curate and oversee as AI developments progress: behavioral benchmarks and knowledge benchmarks.
One example of behavioral benchmark applies game theory to investigate the human traits learned by LLMs (Mei et al. 2024). Over the years, tens of thousands of human subjects in dozens of countries have participated in laboratory experiments examining strategic interactions formalized through game theory. Machines can now play those games too. Relative to machines, humans display greater heterogeneity and are more difficult to predict. This type of reference point is crucial when discussing the deployment of agentic AI across social contexts. Developing further benchmarks will allow us to better understand how LLMs differ from human behavior, while also tracking how human behavior itself adapts and changes as these technologies become widespread.
One example of knowledge benchmark is the recent compilation of expert-level academic questions that help assess AI capabilities and limitations (Phan et al. 2026). This type of reference point relies on a collaborative effort to compile exam questions that require both text and image comprehension across a range of subjects, from math, physics and biology to the humanities and the social sciences. Unlike other existing benchmarks used by AI companies, which are already saturated (and therefore not very useful at measuring performance, already too clustered at the top), the expert-led benchmark reveals LLMs to consistently have very low accuracy. In other words, when experts define the metrics of success, the distance between frontier LLMs capabilities and human expertise is undeniably large. This is the value of questioning corporate-defined measures: it exposes hidden limitations and critical shortcomings that would be otherwise overlooked.
Current benchmarks rely on narrow measures of technical knowledge and reasoning, heavily favoring math and STEM subjects. This imbalance exists partly because the teams developing AI systems are predominantly composed of STEM researchers. Social scientists should play a larger role in defining the reference points we use to evaluate AI’s performance. We need to define the capabilities and failures of these tools not just within research workflows but also across broader social contexts, where norms and institutional arrangements are not always apparent. The truest measure of human performance, in the end, is the ability to follow unwritten rules and honor unspoken promises.