AI Tools Will Transform How We Evaluate Published Science

Written by Roger Pielke Jr on November 4, 2026. Posted in Current News

Last week, economist Tyler Cowen set off a controversy when he shared the results of an AI-assisted audit of research by economists Emmanuel Saez and Gabriel Zucman. Cowen’s post was amplified by Elon Musk on X/Twitter

Saez and Zucman have published influential estimates of increasing concentration of income and wealth in America’s richest households, and they used those estimates to argue for higher taxes on the rich, including a wealth tax — perhaps explaining Musk’s interest in what is otherwise an esoteric debate over academic economics.

According to Cowen, an anonymous auditor(s) used AI tools to examine 138 of Saez and Zucman’s papers and identified potential problems in those papers. The report judged 147 of the problems to be serious and split them into 53 major defects and 94 major contested choices, and concluded:

“No finding in any work was severe enough to overturn its core result outright: the direction of the main results generally survives, while headline magnitudes often do not.”

The anonymous auditor(s) posted the full audit on a public website, but not the underlying code or data.

Zucman responded forcefully on X. He called the audit an “AI hit job” and noted its timing, six days before a vote on California’s billionaire wealth tax.

I don’t have a dog in that fight, but I do find the dispute fascinating for what it signifies for the future of science and, in particular, quality control in published research. The cost of independently checking published research has collapsed — in terms of the time necessary and resources required — and scientists should pay close attention.

Today, I discuss these remarkable changes to set the stage for releasing a significant replication of my own later this week — one that finds major errors in a pair of influential papers on climate economics, errors that are comparable to (or even worse than) those that led to the retraction of Kotz et al. late last year.

A Scientific Paper Already Works as a Prompt

As I have experimented with AI tools in my research, I have come to see a well-written scientific paper as an extraordinarily detailed prompt, perfectly suited for using AI tools in replication.

Consider what a research paper actually does:

The authors state a research question or hypothesis, describe their data, and detail their methods, equations, and assumptions.
They present results, explain how they obtained those results, and offer an interpretation of the significance of those results.
Often (but not always) researchers share their data and code to facilitate replication.

In other words, via publication of a paper authors typically give other researchers enough information to reproduce their findings. In many, if not most, fields of research this practice has long been an expectation, but it is not always followed or enforced by journals.

With the capabilities of AI tools and a well-written paper that follows these norms, a skilled researcher can directly oversee a replication of published work with increasing ease.

Give a capable AI system a paper, its supplementary materials and the data, and ask it to reproduce the findings. The system can translate methods into code, retrieve data, rebuild figures and compare the replicated numbers with the authors’ published numbers.

AI tools also allow for independent verification of an AI-based replication, helping to root out AI-induced errors before the human researcher then evaluates the work.

When discrepancies appear between published work and a replication, the researcher can use AI tools to trace the sources of those differences. Did the authors define a variable in the wrong way? Did they drop observations without explanation? Did they rely on an undocumented assumption?

Artificial intelligence puts human intelligence on intellectual steroids. Expert judgment from a human remains essential and irreplaceable (for now), but the work of coding, data retrieval, data checking, multiple iterations, and more can be undertaken with AI tools.

Of course, authors do not always write a good paper that can serve as a complete and coherent prompt. They also may withhold data, leave decisions undocumented, or describe methods ambiguously, and then any replicator, human or AI, will struggle to successfully replicate the work.

But that too would be an important finding of a replication effort.

With AI tools available to everyone, we now have the capability to perform rigorous replications of published work at unprecedented scale. Buckle up, it is happening.

AI Already Rivals Human Peer Review

Earlier in the same week as the Cowen/Zucman debate flared up, Gary King and Steven Worthington of Harvard University released a fascinating new working paper, “AI Article Reviews Dominate Human Peer Reviews at Their Best: Evidence and Policy Choices”.

King and Worthington compared AI-generated reviews of academic papers with strong human peer reviews: overall the study had 134 paired reviews. Worth noting — in this project peer reviewers were paid for their detailed reviews, something that only rarely happens in real-world academia.

An AI judge preferred the AI review in all 134 comparisons. In a smaller blinded evaluation, former journal editors preferred the AI reviews 87 percent of the time (26 of 30). The authors also counted substantially fewer incorrect claims in the AI reviews.

They conclude that today’s AI systems can outperform even exceptionally capable human peer reviewers.

Scientists have long relied on peer review for quality control, but the process also has notable weaknesses: it can be slow, inconsistent, opaque, and is typically dependent on volunteer experts, who often these days use their political bias in their reviews.

King and Worthington evaluated manuscript reviews, and not replication. Both reviewing and replication, however, require understanding a paper’s methods, evaluating its assumptions, and judging whether the authors have supported their conclusions.

Replication also has a big advantage over peer review: many of its central questions have checkable, quantitative answers. A replicator either reproduces a reported calculation from the stated data and methods or finds a discrepancy.

Human experts still need to judge the significance of each discrepancy, but when AI handles much of the initial, typically quantitative, work, the costs of scientific scrutiny go way down.

A Replication Example from My Research

In 1999, Chris Landsea and I published a paper in the Bulletin of the American Meteorological Society that linked U.S. hurricane damage to the El Niño–Southern Oscillation. Over 1925 to 1997, we found much larger losses in La Niña years than in El Niño years — 2026 is holding to form!

Earlier this year, working with Claude Code, I replicated that paper for a THB Deep Dive report, and I also reproduced its analysis for the years since we published our paper.

The replication followed the published 1999 methods: I classified each year by the August–October Niño 3.4 temperature anomaly and compared normalized damage across El Niño, La Niña and neutral years.

I then applied the method twice, once to the 73 years available to the original paper and once to the 28 hurricane seasons since.

For 1925–1997, the replication confirmed the 1999 findings, and the extension found that the relationship held in the years since our original study.

I also learned something about the details of the replication of older work — datasets change, definitions evolve, and newer work needs to keep up. In updating the work, I made three changes to the 1999 analysis:

NOAA’s current sea-surface temperature record (ERSSTv5) in place of the Kaplan reconstruction,
NOAA’s thresholds of ±0.5 °C in place of our ±0.4 °C,
and the damage normalization of Weinkle and colleagues (2018) in place of ours from 1998.

With those changes, the update reclassified a few marginal years: our 1925–1997 counts came to 23 La Niña, 30 neutral and 20 El Niño years, against 22, 29 and 22 in our 1999 paper.

Replication has some complexities that necessitate human expert judgment that AI tools (so far) cannot replace.

The Coming Avalanche of Replications

For decades, would-be replicators have faced a fundamental problem in doing their work: Scientists are rewarded professionally for publishing original research and receive little professional credit, if any, for checking existing research.

Indeed, the over-the-top response by Zucman to Cowen’s post discussed above clearly indicates that replication can sometimes be met with outright hostility, especially when errors are asserted — we all likely remember the decade-long and extremely nasty public battle over the so-called “hockey stick” in climate science.

Replication can be little fun as well — Replicators must reconstruct datasets, decipher poorly documented code, perhaps rerun analyses in unfamiliar software, and even when they find consequential errors, journals may not have much interest in publishing their findings.

As a result, few people ever scrutinize most published science.

AI tools completely change this equation. When the cost of replicating a paper falls dramatically, researchers can examine many more papers, and they will inevitably find many more problems.

With platforms like Substack now widely used and read, it is much easier to share replicated work and bypass the alleged gatekeepers of science.

Flaws in published research do not imply carelessness or dishonesty among scientists. Scientists, like all people, make mistakes. Research is difficult. Peer reviewers might catch some of these problems, but most of the time peer review is at best just a cursory evaluation of submitted work.

AI-supported replication is going to turn up both genuine errors and false claims of errors. To tell the two apart, replicators must expose their work to the same scrutiny — replications must also be replicable.

In the Cowen-Zucman debate that started this post, I note that the claimed replication by the anonymous auditor(s) did not come with published data or code — for me that is enough to ignore the exercise.

As replication becomes routine, I expect researchers to discover long-overlooked problems in many influential papers — perhaps even in some of my own. I’ve been replicating many of my early papers, and so far so good!

Some authors will need to fix minor errors. Others will need to correct important results. And in a few cases, replicators will overturn findings that may have shaped scientific understandings and public policy.

Scientists should welcome that prospect as a chance to improve the reliability of science. Science would risk a loss of confidence if consequential problems in published work are identified, but the community refused to correct the record. This is not simply speculative, of course.

Recall the fate of the fatally flawed ICAT dataset, which was used in papers that two prominent journals refused to retract, even though the errors are obvious and significant. I lost considerable confidence in journals and editors after that episode.

A Busy Future for Journal Editors

Imagine a world of routine AI-assisted replication. Editors publish an empirical paper. A researcher directs an AI system to reproduce the results, finds a potentially consequential discrepancy and publishes the analysis with data, code and documentation.

Then the editors face a decision.

Under the traditional system, editors might handle the dispute through a letter, a published comment, a correction or, in rare cases, a retraction. That system did not envision large numbers of low-cost replications.

What will editors do when they receive dozens or hundreds of credible replication challenges for papers published years or decades ago?

They will need procedures for identifying genuine errors, judging the significance of discrepancies, getting responses from authors, and for correcting the scientific record. These are not simple challenges.

The scientific community will also need to accept that experts can practice quality control outside the pages of their journals. A researcher who publishes a replication openly, with complete materials, can make an important contribution with or without a peer-reviewed outlet — or a PhD or faculty position.

When anyone can inspect an analysis immediately, editors will find it hard to ignore obvious errors in published research. Ultimately, the coming replication revolution will democratize science.

That also means that trust in science will face new challenges as errors are found and communicated. How researchers, journal editors, and the media respond will go a long way to either reinforcing or compromising public trust in science.

Five THB Rules for AI-Supported Replication

As AI-assisted replication spreads, we need norms for how we should enter this new world of science. I propose five rules for the replications I publish at The Honest Broker — I welcome your comments, critique, and suggestions.

1. Provide all results. Publish every replication, the ones that confirm the original results along with the ones that find errors. If an AI tool-based project examines 100 papers and finds consequential problems in five, readers should know what happened with the other 95. This will help to avoid a version of the so-called “file drawer problem.”

2. Show your work. Share the code, procedures, data sources, model specifications, and other materials that others need to reproduce the replication exercise. Document the models used, how they were instructed, which tools and data they accessed, and how outputs were evaluated. Because AI systems can give different outputs from one session to the next, run the analysis more than once, ideally with independent tools, and report whether the results converge.

3. Publish everything in public. Put the complete analysis and supporting materials in a durable public repository where anyone can inspect, download, and cite the files. A replication should be replicable. Make transparency the default from the start.

4. Give the original authors an opportunity to respond in public. The original authors should be provided the opportunity to examine the replication and to explain or challenge its findings. At THB, I will publish a response from any author of a paper I replicate.

5. Treat peer-reviewed publication as optional. Readers should judge a replication by the quality of its analysis, the availability of its materials, and whether others can verify its findings. A replicator need not wait for peer review before reporting a credible replication publicly — that removes a bottleneck, avoids gatekeeping, and speeds the self-correction of science. This is a lesson I learned the hard way with my dealings with the papers based on the fatally flawed ICAT dataset.

The Future of Scientific Quality Control

In the Cowen–Zucman controversy, we caught an early glimpse of a much larger transformation. We have started to move from a world in which few scientists replicate published research, because they face high costs and little professional reward, to one in which researchers scrutinize published work as a routine part of scientific practice.

This will take place on the page of scientific journals, but I fully expect more commonly it will occur at places like THB on Substack.

Indeed, I fully expect that almost all researchers will engage in AI-assisted replication of their own work before submitting to a journal, which will have the effect of improving the quality of research going forward. Pre-replication — so to speak — should be standard practice in the era of AI tools.

Researchers in climate science, medicine, epidemiology, psychology, political science and many other fields have long relied on large bodies of empirical research without an expectation that their work will undergo systematic replication. With AI tools, we can now examine those literatures, and replication will become common.

Scientists should welcome the coming replication revolution, as uncomfortable as it may be at times.

And journal editors? They should clear their calendars — they are going to be busy.

See more here substack.com

Some bold emphasis added