7  All Models Are Wrong, But Some Are Trending

Mike Burnham, Texas A&M University

Abstract: TBD

AI usage statement: TBD

7.1 Introduction

My core argument is that AI will increase academia’s exposure to the attention economy. It will do so by lowering the cost of producing research and flattening the distribution of credible research. At present, the ability to clear cost and credibility filters helps signal what work is worth paying attention to. When those filters weaken, institutions that rely on them (e.g. the job market, tenure, journalists, and other researchers) will need other ways to decide what deserves attention. As a result, more researchers will compete for attention directly.

This essay focuses on the opportunities and potentially positive outcomes of that development. Giving researchers stronger incentives to seek attention has obvious downsides: clickbait research, publication bias, empowered gatekeepers with access to journals and media outlets, and a greater importance of social networks, to name a few. Those risks deserve attention, but the opportunities are less obvious and will require deliberate action to materialize. I focus on three in particular:

  1. New forms of publications and journals that are more public facing and adopt a form factor beyond the PDF.
  2. Near frictionless publication of traditional research papers in discipline oriented clearing houses.
  3. Significant revisions to tenure and hiring standards.

I first explain how AI changes the cost and credibility of research and why that creates stronger incentives to compete for attention. I then outline several potentially positive developments that could follow.^[Forecasts about AI usually rest on implicit assumptions about how far capabilities will advance and which tasks cannot be reduced to computation. Those who imagine a future where their current job is done alongside, rather than by, AI assume model intelligence will not dramatically outpace their own, that abstract qualities like “taste” are non-computable, or at least that there is a normal amount of paper clips in the world.

For present purposes, I assume there is some upper bound on model intelligence around human capability, that humans possess some irreducible qualities, that agents remain agents, and that humans remain principals. I have no intention of defending these assumptions. They are not the assumptions I think are most likely; they are simply the ones that produce a world familiar enough to make predictions about. As for what assumptions about AI progress I find most likely, I have no clue.]

7.2 AI Will Flatten Cost and Credibility

Cost and credibility are two axes along which research production faces important filters. These filters constrain both the volume and the types of projects that move beyond the ideation stage. A project’s ability to clear them serves as a noisy signal of what work is worth attention. What follows is a brief account of the filters AI is likely to weaken and how that shift could reshape incentives and opportunities for researchers. This is not an exhaustive list, nor a full defense of each claim. It is an outline of the premises behind my argument that AI will flatten the cost and credibility distributions of research.

% One of the primary effects of AI on social science will be to alleviate some of these filters. This both disrupts the signals currently used and presents new opportunities. Lower data collection costs mean more researchers can attempt projects that previously required grants, teams of assistants, or institutional infrastructure. Lower software development costs mean more researchers can build the tools, archives, and interfaces that make a project both useful and attention-grabbing. On the credibility side, better access to robustness checks, criticism, and technical review means the modal paper may become more credible, or at least communicate a clearer credibility signal. In other words, AI may not just give us more papers. It may change the baseline level of polish and scrutiny that a paper receives before anyone else sees it.

7.2.1 Cost Filters

I want to distinguish the claim that AI will reduce the cost of producing research from the claim that it will increase the supply of research. The latter seems likely to me, and may already be happening, but it does not necessarily follow in the long run. AI will affect research designs unevenly. While the cost of observational studies built from scraped web data may fall sharply, experiments on human subjects may be far less affected. Researchers may therefore seek comparative advantage in slower and costlier designs. Similarly, I am only slightly joking when I say that Parkinson’s law remains undefeated, and we may not get more publications unless we also get more conference deadlines to plan around. More charitably, researchers may choose to invest AI’s labor-saving effects in better work rather than simply in more work.

The point is that cost filters both volume and quality. Reducing cost makes it harder to judge which projects deserve attention, either because there is more work, quality is more evenly distributed, or both.

7.2.1.1 Data collection

AI agents are highly capable of autonomously scraping the web, querying APIs, extracting data from text, and managing the end-to-end data cleaning pipeline. They are increasingly used to conduct phone surveys, run simulations, and sample in silico for pilot studies. AI-led data collection will only get more reliable and cost-effective as researchers engineer harnesses to execute these activities in reliable and reproducible ways.

7.2.1.2 Software development

Software development has become a normal part of social science publication. Often this is just a replication archive; sometimes it includes software packages or executable programs. But social scientists are not usually trained as software developers, and even simple code bases can become major time sinks or require hired help. For many researchers, project- or lab-specific websites that host web apps and interactive data sets such as Voteview (Voteview 2026) or the Polarization Research Lab (The Polarization Research Lab 2026) were not realistic. Now, agentic coders can write scripts in minutes and build research-relevant web apps in hours.1

7.2.2 Credibility Filters

Journal quality and professional reputation are the primary signals researchers use to determine what work is worth attention. If the bottlenecks to producing and certifying credible research are eased, credible work will become both more abundant and more evenly distributed across journals and researchers. Journal placement will still matter, but it may no longer monopolize credibility in the same way. More work will clear basic thresholds of competence, and it will become harder for researchers to distinguish themselves through rigor alone. In that world, attention becomes more contestable. Researchers have stronger incentives to compete for it directly because the traditional filters do relatively less work. I see three mechanisms through which AI could weaken the credibility filter.

7.2.2.1 Robustness checks

The ability of AI agents to rapidly prototype methods and test modeling assumptions has been, perhaps, the most significant change in my own research. In Gelman and Loken’s (Gelman and Loken 2013) “Garden of Forking Paths”, they show how researcher degrees of freedom can generate false positives even without any intent to p-hack. Researchers make many decisions in data collection, cleaning, variable construction, and analysis that affect their findings. Those decisions are often reasonable and defensible, but alternative choices may be equally defensible and yield different results. Robustness checks are, in part, an effort to explore those alternative paths. Done manually, that process is often laborious and expensive. AI agents that automate large parts of data collection and analysis could make it much easier to present a fuller “garden of reasonable paths.”

7.2.2.2 Early stage feedback

Early in my Ph.D. program, I realized that the most valuable resource in academia is criticism—and that it is very unequally distributed. The most well-known researchers sit on the most panels, receive more invitations to present their work, and see their preprints discussed more widely. That is more observation than complaint. However, producing credible research is harder when critical feedback is scarce and the unequal distribution of critical feedback has downstream affects on the distribution of credible research. Language models show some promise in democratizing early critical feedback before work reaches peer review (Refine 2026) This is undoubtedly a benefit, but it does imply that reputation will be more difficult to establish and provide a noisier signal in the future.

AI agents that can inspect code bases, run robustness checks, and critique entire papers could make peer review more exhaustive than it currently is. My claim is not that AI will replace peer review entirely, though I do not rule that possibility out. It is simply that many journals do not replicate papers, and reviewers rarely examine code bases in depth. Robustness checks are subject to the whims of reviewers and cost or feasibility constraints. Agentic reviewers could improve both the depth and consistency of this process at low cost. A more exhaustive paper and code review might be completed on the day a paper is submitted rather than after months of back-and-forth. I expect journals to build agentic review harnesses in standardized ways and for AI review to proliferate on preprint repositories like arXiv and OSF. Wide proliferation of quality reviews implies that traditional signals of quality and the journal level will be less informative.

7.3 Opportunities

Consider two facts that I think are tenuously related:

  1. On April 4, 2026, the National Science Foundation announced that it would dissolve the Social, Behavioral, and Economic Sciences Directorate in response to budget requests from the Trump administration (Kozlov, Garisto, and Chen 2026)
  2. First-year Ph.D. students are often taken aback by the number of statistics courses they are required to take and express frustration that this is not what they meant to sign up for.

I want to be clear that I do not think social scientists are to blame for recent funding cuts. But I do suspect that broad misperceptions of 8what social scientists actually do is at least a small contributing factor. In my experience, most people *with college degrees cannot distinguish between the humanities and social sciences. I regularly talk with academics in other fields who imagine social scientists as reading dusty tomes and writing essays while smoking a pipe and gazing over the Potomac. If even undergraduates who plan to pursue Ph.D.s often misunderstand the work, then broader society is unlikely to understand it either. And if people do not know what we do, it is harder to expect them to value and fund it.

My point is this: social science may benefit from a greater pursuit of attention. Most discussions of the attention economy understandably focus on its pathologies, and I do not want to dismiss those risks. If attention becomes more valuable, some researchers will pursue it in ways that do not advance knowledge. But the same shift may also create an opportunity to build institutions that communicate our value more clearly to the public. I do not expect good institutions to emerge automatically from the incentive change. In one future, research devolves into top 10 Twitter threads you won’t believe, complete with siren emojis. In another, we build institutions that incentivize and reward healthier forms of attention competition. We should seek to build institutions that have greater public reach while still producing valuable research. Reaching the better future will require deliberate effort.

What might those institutions look like? Here I think we have considerable agency. The proposals below are meant as starting points, not predictions.

7.3.1 New forms of publications and journals

For most research projects, publication is a funeral. Academia offers few incentives to keep research alive after publication. For many projects that is fine: a question has been answered, and the authors have nothing more to say. For others, it is a loss. Living publications can be both a scientific and a public good. Voteview is perhaps the clearest example. The site, which hosts DW-NOMINATE ideology scores for Congress, has had an enormous impact on both scholars and journalists. It provides current data, public tests of predictive validity, and simple, easy-to-use interfaces.

Not all research needs life after publication, but Voteview is not unique in being able to benefit from it. Models that can be continually validated against new events, along with measurement tools, descriptive data, and simulations that can be updated as new data is generated from the real world, have substantial scientific and public value. They create opportunities to test the predictive validity of earlier work, enable new research, and show the public what our work contributes.

The relative scarcity of such living publications is due partly to weak incentives and partly to the technical hurdles involved in building and maintaining them. Tenure guidelines do not reward useful websites even when they are genuine public goods. Until recently, the technical and financial costs were often hard to justify. That may now change. The technical hurdles are falling, and the attention rewards from living publications may become more attractive to researchers.

These changes may manifest in the form of disparate web apps on personal pages. That would be a tragedy. We should instead build publication outlets that are both better suited to dynamic form factors and more explicitly public-facing. We should also reward such work on the career track. If researchers begin competing more directly for attention, then journals that help readers allocate attention efficiently may become more valuable than journals that merely withhold publication.

7.3.2 Frictionless publication

The second opportunity is near-frictionless publication of more conventional papers. If AI review becomes a legitimate competitor to peer review as a credibility signal, then the incentive to spend months or years navigating conferences and long review cycles may weaken. Consider how publication has evolved in NLP and AI. Prestige outlets have not vanished; in some ways they matter more than ever. But the circulation of ideas no longer waits for them. Preprint servers, code repositories, and public benchmarks have become part of the infrastructure through which the field thinks in real time. This has worked in AI partly because credibility is often easier to evaluate: the engineering works or it does not. Social science will remain costlier to validate, especially when human subjects are involved, but in some areas those costs may fall substantially. If AI systems make replications, code inspection, reporting checks, and robustness tests cheaper, then the credibility gap between a top journal and an open repository may narrow. More researchers may then want to release work quickly in open repositories alongside AI reviews, especially when the goal is visibility, citability, and response.

We should actively facilitate this shift. Institutionally, that might look like something closer to arXiv, organized by disciplines or professional societies, with lightweight submission requirements and standardized automated review. On submission, an AI review system could check documentation, attempt replication, summarize robustness concerns, and generate a public report. This would not create a gold standard of truth. It would, however, create a low-friction way to make work public while giving readers a clearer signal about credibility. To much of the public, an AI-generated review may be more informative than knowing that the paper was published in the American Political Science Review.

For some projects, such publication would be the end state; for others, it would be the beginning. Platforms like this would also create new roles for curators. Editors, review boards, and disciplinary associations could still decide what deserves amplification, special issues, invited symposia, or formal publication.

7.3.3 New tenure and hiring standards

If the media of publications diversify, tenure and hiring standards must change as well. Current institutions have enormous power to shape researchers’ incentives, so they should tread carefully. The obvious danger is to make attention itself the metric, but I trust we can avoid a world in which academics count clicks alongside citations.

Still, the answer cannot be to pretend the old signals remain as informative as before. How should we reward the continued maintenance of living publications? How should we value AI-reviewed versus peer-reviewed work? We need a broader account of scholarly contribution, but one that still allows concrete evaluation. That may mean giving more credit to publicly useful data sets, software tools, replication packages, synthetic reviews, interactive publications, and forms of disciplinary service that improve the credibility and accessibility of research. It may also mean paying more attention to whether a scholar’s work matters beyond academia.

7.4 Conclusion

In some ways, these changes would be a return to first principles. The purpose of tenure, hiring, and publication is not to reward successful navigation of unnecessary frictions. It is to identify people and work likely to produce valuable knowledge, contribute to a discipline, and improve our institutions. If AI changes how knowledge is produced and validated, then our standards should change too. Ideally, they should resist pure attention-seeking while still recognizing that public communication, tool building, and open scientific infrastructure are genuine scholarly goods. If the incentive structure is going to shift regardless, the goal should not be to stop it. It should be to reward the versions of that shift that make the discipline more useful and more legible.

  1. See, for example, this supreme court composition simulator that was vibe coded with Claude Code: https://jkastellec.github.io/msc-simulator/.↩︎