AI know-how was alleged to streamline scientific peer assessment. As an alternative, it’s proving simple to idiot.
For roughly a decade, new analysis papers have piled up quicker than scientists can rigorously assessment them. Some researchers have resorted to AI instruments to lighten their reviewing load, chopping time spent on paper evaluations from days or even weeks to minutes. However scientists can simply manipulate their papers to trick an AI peer assessment instrument into score them as stronger or extra publishable than they are surely, pc scientist Joachim Baumann and colleagues report. What’s extra, AI-generated evaluations typically sound the identical, shedding the nuance and variety of human analysis.
“We’re being swamped with extra papers than we’ve the capability to assessment, so we do want some options, and automation will help for some elements of it,” says Baumann, of Stanford College. However thorough experiments and analysis are wanted earlier than such instruments enter the peer assessment course of, he says. In any other case, AI instruments would possibly inadvertently perpetuate the biases they’re identified to hold and scale back the number of opinions weighing in on new science.
The workforce will current their findings July 8 on the Worldwide Convention on Machine Studying in Seoul, South Korea.
Many researchers have already adopted AI instruments of their work. Of the practically 20,000 papers submitted to the 2026 Worldwide Convention on Studying Representations, or ICLR, about 1 in 5 have been totally AI-generated, in keeping with a case examine from November by the corporate Pangram. AI is changing into frequent in peer assessment, too. A December survey of 1,600 scientists in 111 international locations discovered that greater than half had used AI instruments to assist assessment papers, together with summarizing research and assessing the energy of a paper’s arguments.
“AI instruments are inherently opaque and dilute obligations and accountabilities,” says Mohammad Hosseini, a bioethicist from Northwestern College Feinberg Faculty of Medication in Chicago who was not concerned in both examine. “If you introduce a nontransparent actor like AI inside a system that for a very long time was attempting to turn into extra clear, it’s a step backward, and there could be unexpected penalties.”
One consequence might be a loss within the variety of suggestions on manuscripts. Range of opinions is vital in paper evaluations “as a result of a whole lot of these selections of whether or not to just accept the paper or not, whether or not one thing is novel sufficient or not, whether or not a sure limitation is sufficient to have a paper rejected, these are sometimes very subjective selections,” Baumann says. “It appears pure that we additionally need a numerous set of opinions to be represented every time we automate issues with AI.”
Within the examine, Baumann and his colleagues analyzed AI-generated and human-written evaluations of papers submitted to ICLR 2026. The workforce examined the semantic and linguistic patterns within the evaluations and located that these generated utilizing AI instruments have been way more much like each other than human or human-assisted evaluations.
The researchers additionally randomly chosen 60 ICLR papers and prompted AI fashions to generate detailed evaluations within the method of a human reviewer at ICLR. Then they requested two giant language fashions to rewrite the papers to acquire greater scores primarily based on the suggestions within the AI-generated evaluations. Typically, the scores given by three AI reviewer fashions after the rewrite have been greater than from the AI-generated evaluations earlier than.
Many of the modifications made through the rewrite have been stylistic, similar to the usage of hedging phrases like “could” and “suggests” and emphasis phrases like “robust” and “strong.” A few of these adjustments may need made issues clearer, however there have been additionally apparent circumstances of scientific misconduct, Baumann says. Fashions added findings from experiments that weren’t truly run, in essence making up outcomes, he says. AI-generated evaluations of those 60 papers have been additionally much more much like one aside from the human evaluations have been, each earlier than and after the rewrite, for a similar paper.
Many conferences now prohibit the usage of AI instruments for peer assessment. Others are experimenting with and evaluating the standard of AI-generated evaluations to find out whether or not AI ought to be formally built-in into the assessment course of. However whereas efficiency on some duties like checking for hallucinated references and formatting errors could be simply examined, subjective questions on whether or not a paper’s contribution is significant to a analysis group are a lot more durable to guage, Baumann says.
He and different researchers ponder whether AI reviewers would have the ability to choose new analysis that goes towards prior work or introduces one thing novel, similar to a brand new experimental setup or a brand new mannequin structure. “There simply is perhaps sure subjects that get a low rating from AI reviewers, despite the fact that they might be extremely invaluable contributions to the group,” Baumann says.
Their analysis additionally discovered that the 60 rewritten papers have been way more comparable to one another than the unique papers have been. There’s a priority that this might result in an “mental monoculture,” the researchers write. If, for example, many researchers use the identical giant language mannequin to assist them write the paper, there can be extra comparable papers, and “scientific writing will converge towards no matter type the AI reviewer rewards,” the workforce writes.
Whereas it is a critical danger, it won’t be one unique to AI reviewing scientific papers, says Graham Neubig of Carnegie Mellon College in Pittsburgh. “Paper authors have lengthy thought of ‘what is going to reviewers assume’ once they write papers, and this may trigger them to go for ‘safer,’ extra incremental concepts and noncontroversial subjects,” he says. “In a method, AI-enhanced assessment processes could even present a approach to push again towards this, by explicitly encouraging AI reviewers to reward extra inventive concepts.”
