二次重述的批评:LLM 时代同行评审中主张层面的同质化

媒体报道 Hugging Face Papers 原文语言:中文

作者分析了各大AI会议和期刊同行评审中批评意见的重合度,发现 ICLR 的评审重合度从 2026 年的 12.0% 上升到 2027 年的 21.3%。研究通过大模型文本检测器(MELD)指出,带有 LLM 生成特征的同行评审之间比未标记的评审具有更高的重合度,暗示同行评审的批评多样性正在下降。

全文

Twice-Told Critiques: Claim-Level Homogenization of Peer Review in the LLM Era

Ilya Gusev Affiliation: Keenable / Amsterdam Email: phoenixilya@gmail.com

Abstract

Peer review relies on several readers finding different problems. We measure critique overlap in conference and journal reviews. At ICLR, measured overlap rose from 12.0% in 2026 to 21.3% in 2027, in reviews from the same stage of the review process, with year-on-year rises at NeurIPS, ICML and the F1000 and PLOS journals. A third ICLR review of a paper added 5.46 new problems to the other two in 2027, against 6.12 in 2026. We apply MELD, a detector of LLM-generated text, to all reviews. Within a paper, two reviews that MELD flags overlap more than two unflagged reviews: by 3.0 points at ICLR in 2024–2025, by 4.5 points in 2026–2027, and by 5.4 points across five journal sources. These results provide evidence that LLM-flagged reviews of one paper are less distinct from each other than unflagged reviews, and suggest that the diversity of critiques among reviews of a single paper is falling.

Figure 1: Measured critique overlap rose at ICLR in 2027, at ICML in 2026 and at NeurIPS in 2025. Journal overlap rose most at F1000 and PLOS. We define overlap as shared problems divided by the smaller critique set, averaged within papers, then over papers. Dashed segment: from 2026 on, ICLR samples hold reviews during the discussion phase. The 2026 and 2027 points include papers withdrawn later.

1 Introduction

Peer review needs readers who find different problems. In one study, two reviewers usually shared no critical point (Fiske and Fogg, 1990). In another study about Nature-family reviews, 25.8% of one reviewer’s criticisms match another’s (Kim et al., 2026a). Our central question is: with widespread LLM-generated text in ML reviews (Liang et al., 2024a; Russo et al., 2025; Emi, 2025), do reviewers still provide distinct critiques?

We measure shared problems by extracting and matching critiques across reviews of one paper. Whole-review similarity mixes wording and content (Liang et al., 2024a; Baumann et al., 2026) and cannot establish that two reviews raise the same problem. Prior studies find that synthetic LLM-generated reviews are more similar to each other than human reviews (Hou et al., 2026) and share more criticisms (Kim et al., 2026a), a gap used for detection (Duarte et al., 2026). We apply critique extraction (Liang et al., 2024b; Kim et al., 2026a) to reviews that venues received, comparing problems within papers and over time.

Our findings are:

•

Critique overlap rose at ICLR from 12.0% in 2026 to 21.3% in 2027, at ICML in 2026 and at NeurIPS in 2025 (Section 4, Figure 1).

•

Within a paper, two LLM-flagged reviews overlap more than two unflagged reviews at ICLR, by 3.0 points in 2024–2025 and 4.5 points in 2026–2027 (Section 4, Figure 2).

•

Pooled over five journal sources, the same within-paper gap is 5.4 points (Section 4, Figure 4).

2 Related Work

LLM use in peer review.

ICLR and NeurIPS reviews grew longer after ChatGPT, while LLM-assisted ICLR reviews mention originality less (Wu et al., 2026). LLM reviewers draw more than half of their weaknesses from the appendix. In sampled ICLR reviews, the share of such weaknesses rose from 11.3% in 2026 to 27.6% in 2027 (Ghosh et al., 2026). LLM-generated reviews at one management journal cover a narrower range of topics (Gartenberg et al., 2026). At ICML 2026, a randomized trial assigned papers and reviewers to a policy that prohibited LLM use or to one that allowed limited use. The second policy made reviews 5.5–7% longer and left scores and decisions almost unchanged (Kim et al., 2026b). We compare shared problems, not single-review topics.

Homogenization.

LLM assistance makes the texts and ideas of different users more alike (Padmakumar and He, 2024; Anderson et al., 2024; Doshi and Hauser, 2024). Model outputs are narrower than training data (Wu et al., 2025). Sourati et al. (2026) review this evidence. We extend the question from creative tasks to expert critiques, where independent readings are the goal.

3 Methodology

3.1 Data

Table 1 lists the samples. ICLR and NeurIPS come from OpenReview (ulab-ai, 2025; Xu et al., 2025; Heineman, 2025). ICML includes accepted papers from 2025–2026. ICLR 2027 reviews come from the OpenReview API eight days after they became public. Conference papers need at least three reviews. Journal papers need at least two first-round reviews: SciPost, F1000Research and sister platforms (F1000), PLOS, BMC Medicine (JerMa88, 2025), and Copernicus/EGU, covering physics, medicine, biology and geoscience. A panel is the set of reviews of one paper.

Source Years Papers Reviews

ICLR 2020–27 85,466 299,168

NeurIPS 2021–25 18,746 74,866

ICML 2025–26 9,588 36,804

SciPost 2019–26 2,962 6,613

F1000 2019–26 7,111 16,983

PLOS 2019–26 14,360 34,084

BMC 2019–25 2,510 6,414

Copernicus 2019–26 16,000 34,877

Table 1: Papers and reviews in the analysis.

3.2 Claim extraction

A critique claim identifies one problem that needs one fix. We exclude praise and neutral text. In pass 1, GPT-6 Luna (Luna) reads one review in full and lists its claims. Each claim has a quote copied from the review, a one-sentence paraphrase and a category such as clarity or experiment scope.

We check pass 1 against human labels and against other LLMs. For the human check, we use two public sets: 218 Nature-family reviews that the authors of the set split into 1,930 critique items (Kim et al., 2026a), and 60 NeurIPS reviews with 168 weakness spans that experts marked (Ou et al., 2025). A quote of at least one Luna claim falls in 96.7% of the items and in 94.2% of the spans, and 0.7% of Luna’s claims on the first set fall outside every item. For the LLM check, Sol 6.1, DeepSeek V4.1 Flash and Kimi K3.1 Pro run pass 1 on 2106 of our reviews, in 510 validation panels from every venue, year and detector stratum. We call these three models the council. Luna’s quotes reach a word-span F1 of 78.5% with a council member, against 80.5% between two members.

3.3 Claim grouping

In pass 2, a separate Luna call groups the claims of one panel into problems. The call sees the claim lists, not the review texts. Two claims join one group only when they state the same problem in the same paper element.

To check pass 2, each council member groups Luna’s claim lists with the same prompt. Against the majority vote of the three members on each claim pair, Luna’s grouping has 83.6% precision and 78.4% recall across 507 panels. With a council member or the majority as grouper, the within-paper flagged-pair gap of RQ2 is 7.0 to 7.3 points, against 7.4 with Luna (Table 2). Each council member also runs both passes on its own claims, and all four pipelines give every contrast the same sign (Table 3).

Within-paper gap (points)

Grouper One flagged Two flagged

GPT-6 Luna +2.9 [0.2, 5.6] +7.4 [3.6, 11.3]

Sol 6.1 +2.1 [−-0.7, 4.9] +7.1 [3.3, 10.9]

DeepSeek V4.1 Flash +1.8 [−-1.8, 5.4] +7.3 [2.7, 11.8]

Kimi K3.1 Pro +2.7 [−-0.8, 6.1] +7.2 [2.7, 11.7]

Council majority +2.1 [−-1.3, 5.5] +7.0 [2.7, 11.2]

Table 2: Every grouper gives about the same within-paper gaps. Each grouper groups Luna’s claims on 405 validation panels from 2024 on. Gap: overlap of pairs with one or two flagged reviews minus unflagged pairs of the same paper, with 95% intervals.

Overlap (%) Contrast (points)

Pipeline Same paper Cross paper Flagged gap ICLR 2027 −- 2026

GPT-6 Luna 11.1 0.4 +7.4 +7.0

Sol 6.1 10.9 0.5 +6.1 +9.0

DeepSeek V4.1 Flash 14.8 0.5 +4.2 +7.9

Kimi K3.1 Pro 14.7 1.0 +10.0 +9.6

Table 3: All four pipelines give every contrast the same sign; the overlap level depends on the model. Each model extracts and groups its own claims on 510 same-paper and 90 cross-paper validation panels. Flagged gap: the two-flagged gap of Table 2.

3.4 Metric

Let GAG_{A} and GBG_{B} be the sets of distinct problems that reviews AA and BB raise. Their critique overlap is the overlap coefficient (Simpson, 1960)

o⁡(A,B)=|GA∩GB|min⁡(|GA|,|GB|),o(A,B)=\frac{|G_{A}\cap G_{B}|}{\min(|G_{A}|,|G_{B}|)}, (1)

and o⁡(A,B)=0o(A,B)=0 for an empty set. Full overlap means containment of the smaller set, not identical reviews. Higher overlap can result from smaller critique sets rather than more shared problems. Every main claim keeps its sign under other measures, including shared-problem counts at fixed set sizes. Following the IntraSim measure of Baumann et al. (2026), we average oo over all review pairs within a paper, then over papers within a year.

3.5 Detector of LLM-generated texts

MELD (Li et al., 2026) scores reviews for LLM-generated or edited text. LLM-flagged means above one common threshold: the 99.9th percentile of 110,080 pre-2023 scores. MELD flags at most 0.4% of each source’s pre-2023 reviews and 100.0% of 5,000 generated ICLR reviews (Yu et al., 2026). MELD flags 39.5% of Luna-polished human reviews.

Figure 2: In both ICLR periods, at NeurIPS and at ICML, two LLM-flagged reviews of one paper overlap more than two unflagged reviews. A pair type counts the flagged reviews of a pair. (a) Mean over papers of pair overlap within each paper. (b) Mean over papers of the difference from the unflagged pairs of the same paper. Bars: 95% intervals.

4 Results

RQ1: Do reviewers of one paper repeat more of each other’s critiques?

ICLR overlap rose from 12.0% in 2026 to 21.3% in 2027, by +9.3+9.3 points. These two years are directly comparable: both hold reviews during the discussion phase. NeurIPS and ICML also rose in their latest cycles (Figure 1).

Table 4 counts claims in all three-review subsets of each ICLR paper. From 2026 to 2027, claims per review rose from 7.2 to 7.3, but distinct claims per three-review panel fell from 19.9 to 19.0, and the new claims that a third review added fell from 6.12 to 5.46.

Claims

Year Per review Distinct New in a third review Shared per pair

2020 7.02 19.73 6.16 0.48

2021 7.65 21.62 6.79 0.48

2022 7.57 21.32 6.68 0.49

2023 7.06 19.79 6.17 0.49

2024 6.87 19.44 6.11 0.42

2025 7.20 20.34 6.38 0.45

2026† 7.20 19.92 6.12 0.61

2027† 7.31 18.98 5.46 1.10

Table 4: ICLR three-review panels hold fewer distinct claims in 2027 than in any year from 2020, although claims per review did not fall from 2026. Distinct: claims of the three reviews, with a repeated claim counted once. †Reviews during the discussion phase; earlier years hold final reviews.

RQ2: Do LLM-flagged reviews share a larger fraction of their critiques?

For each paper with both pair types, we subtract the mean overlap of its pairs of two unflagged reviews from the mean overlap of its pairs of two flagged reviews. The flagged-pair gap is the mean of this difference over papers. At ICLR, the flagged-pair gap is 3.0 points in 2024–2025 and 4.5 points in 2026–2027. We do not pool them: their reviews come from different review stages. The gap is also positive at NeurIPS and at ICML (Figure 2).

If the overlap of every ICLR pair had risen from 2026 to 2027 as much as the overlap of two unflagged reviews, from 10.8% to 14.5%, the total would have risen by 3.7 points, not 9.3. The other 5.6 points go with flagged reviews. This split is an accounting, not an effect of LLM use.

The gap could come from wording and not from content, if two reviews in one LLM style are easier to match. To test this style effect, Luna changes two human reviews in each of 600 pre-2023 ICLR panels and keeps their points. Polishing the language changes overlap by +0.5+0.5 points, and rewriting each review from an outline of its points by +0.9+0.9 points, against ICLR flagged-pair gaps of 3.0 and 4.5 points. Style effects can therefore explain only part of the flagged-pair gap.

The gap could also come from the detector and not from LLM use, if MELD gives high scores to reviewers who raise common critiques with or without an LLM. Such reviewers also wrote before ChatGPT. In a pre-ChatGPT check, we flag the top 45% of MELD scores in ICLR 2020–2023, a share close to the 2026 flag rate, and repeat the comparison. The gap is −3.3-3.3 points in 2020, −1.2-1.2 in 2021–2022 and 0.20.2 in 2023 (Figure 3). A high detector score alone does not produce the gap. GigaCheck (Tolstykh et al., 2026a), which flags fewer reviews, gives gaps of 2.2 and 5.4 points at ICLR, 5.1 at NeurIPS and 5.5 at ICML.

Figure 3: The within-paper flagged-pair gap is positive in both ICLR periods, at NeurIPS and at ICML, but not before ChatGPT, and shrinks at lower thresholds in ICLR 2024–2025. FE: paper fixed effects, without and with length and claim-count controls. 95% intervals cluster by paper. Dotted lines: main estimates. Before ChatGPT, we flag the top 45% of MELD scores.

RQ3: Do LLM-flagged journal reviews also repeat more critiques?

From the 2019–2023 mean to 2026, journal overlap rose most at F1000 (+5.8+5.8 points) and PLOS (+2.5+2.5), and at both sources it rose in each year from 2024 (Figure 1). Within a paper, two flagged reviews overlap 5.4 points more than two unflagged reviews. Few journal papers have four reviews, so few have both pair types. For the journals, we therefore estimate the gap with a regression with paper fixed effects, which also uses papers with three reviews and mixed flags. The gap is positive at F1000, PLOS and Copernicus, and the intervals of BMC Medicine and SciPost include zero (Figure 4).

Figure 4: The journal flagged-pair gap is positive pooled and at F1000, PLOS and Copernicus. Two-flagged minus unflagged overlap from a regression with paper fixed effects, from 2024 on, with 95% intervals. Parentheses: papers with more than one pair type. Dark blue line: ICLR 2026–2027, also first-round reviews.

5 Conclusion

Our results suggest that the reviews of one paper have lost part of their independence, in content and not only in wording. Reviews that the detector flags are one part of this change and do not explain all of it, because overlap also rose between reviews that the detector does not flag. Two questions stay open: how much of that rise comes from LLM use that leaves no trace in the text, and whether the 2027 level holds after the discussion phase.

Limitations

LLM measurement. An LLM extracts and groups claims. The council reference uses LLMs, not humans, and does not rule out shared model errors. The human check covers pass 1 only. Extraction and grouping errors can affect measured levels and changes over time. The overlap level depends on the grouper: council members give 10.9 to 14.8% against Luna’s 11.1%. Luna misses more links in papers with no flagged review. LLM groupers agree that this error changes comparisons between papers but not within-paper gaps.

Review snapshots. ICLR samples hold final reviews before 2026 and reviews during the discussion phase from 2026 on, so the ICLR series compares reviews of one stage only from 2026 to 2027. For the same reason, we report ICLR flagged-pair gaps for 2024–2025 and 2026–2027 apart.

Detector labels. MELD trained on peer reviews from sources it does not name. If these include our pre-2023 reviews, the threshold is too low and later flag rates too high. GigaCheck flags fewer recent reviews than MELD, so flag rates depend on the detector. MELD flags 39.5% of our polished reviews, so a flag does not separate polishing from generation. Other tools and prompts can polish differently, and detectors label part of LLM-polished reviews as AI-generated (Saha et al., 2026). A flag does not tell which model wrote a review. Critique content differs between model versions and families (Guo, 2026), so the flagged-pair gap can change with the mix of models that reviewers use. The rewrite control uses two models, two prompts and pre-2023 reviews. Within-paper comparisons remove paper-level differences, not reviewer-level ones, and do not assign a causal effect to LLM use. An unflagged review is not thereby human-written. Detectors can err across language backgrounds, fields and templates. Their labels must not support action against an individual reviewer.

Selection and scope. NeurIPS and most journals publish accepted-paper reviews only, and our ICML sample has accepted papers only. The journal gap rests on 2,228 papers with three or more reviews and mixed flags, and its regression assumes that each pair type shifts overlap by the same amount in every paper. All texts are in English, and most come from ML.

Use of LLM Assistants

The authors created the idea and the outlines of the experiments, and acted as supervisors and editors. Claude Fable 5.2 inside Claude Code and GPT-6 Astra inside Pi wrote all code and all text of this paper. The authors take full responsibility for the content.

References

Anderson et al. (2024) Barrett R. Anderson, Jash Hemant Shah, and Max Kreminski. 2024. Homogenization effects of large language models on human creative ideation. In Proceedings of the 16th Conference on Creativity & Cognition, pages 413–425. Association for Computing Machinery.

Baumann et al. (2026) Joachim Baumann, Jiaxin Pei, Sanmi Koyejo, and Dirk Hovy. 2026. Position: Stop automating peer review without rigorous evaluation. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research, pages 169349–169380. PMLR.

Doshi and Hauser (2024) Anil R. Doshi and Oliver P. Hauser. 2024. Generative AI enhances individual creativity but reduces the collective diversity of novel content. Science Advances, 10(28):eadn5290.

Duarte et al. (2026) André V. Duarte, Brian Tufts, Aditya Oke, Fei Fang, Arlindo L. Oliveira, and Lei Li. 2026. Sem-Detect: Semantic level detection of AI generated peer-reviews. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research, pages 27089–27119. PMLR.

Emi (2025) Bradley Emi. 2025. Pangram predicts 21% of ICLR reviews are AI-generated. Pangram Labs blog, https://www.pangram.com/blog/pangram-predicts-21-of-iclr-reviews-are-ai-generated.

Fiske and Fogg (1990) Donald W. Fiske and Louis Fogg. 1990. But the reviewers are making different criticisms of my paper! Diversity and uniqueness in reviewer comments. American Psychologist, 45(5):591–598.

Gartenberg et al. (2026) Claudine Gartenberg, Sharique Hasan, Alex Murray, and Lamar Pierce. 2026. More versus better: Artificial intelligence, incentives, and the emerging crisis in peer review. Organization Science, 37(3):795–812.

Ghosh et al. (2026) Saptarshi Ghosh, Mohammad Saim, and Tianyu Jiang. 2026. AI reviews are biased towards the appendix. Preprint, arXiv:2611.17811.

Guo (2026) Jizhou Guo. 2026. Same papers, different verdicts: Tracing version drift in LLM peer review. Preprint, arXiv:2611.20608.

Heineman (2025) David Heineman. 2025. iclr-2026. Hugging Face dataset. https://huggingface.co/datasets/davidheineman/iclr-2026.

Hou et al. (2026) Xuzhe Hou, Shengxin Zhang, Riley Stacy, Emily Black, and Varun Chandrasekaran. 2026. Do AI reviewers converge? Diversity collapse in model-generated peer review. In Findings of the Association for Computational Linguistics: EMNLP 2026, pages 21856–21885, Budapest, Hungary. Association for Computational Linguistics.

ICLR 2026 Program Chairs (2025) ICLR 2026 Program Chairs. 2025. ICLR 2026 response to security incident. ICLR Blog, https://blog.iclr.cc/2025/12/03/iclr-2026-response-to-security-incident/.

ICML 2025 Program Chairs (2025) ICML 2025 Program Chairs. 2025. ICML 2025 reviewer instructions. https://icml.cc/Conferences/2025/ReviewerInstructions.

ICML 2026 Program Chairs (2026a) ICML 2026 Program Chairs. 2026a. ICML 2026 policy for LLM use in reviewing. https://icml.cc/Conferences/2026/LLM-Policy.

ICML 2026 Program Chairs (2026b) ICML 2026 Program Chairs. 2026b. ICML 2026 reviewer instructions. https://icml.cc/Conferences/2026/ReviewerInstructions.

JerMa88 (2025) JerMa88. 2025. BMC Medicine peer reviews. Hugging Face dataset. https://huggingface.co/datasets/JerMa88/BMC_Medicine_Peer_Reviews.

Kim et al. (2026a) Seungone Kim, Dongkeun Yoon, Kiril Gashteovski, Juyoung Suk, Jinheon Baek, Pranjal Aggarwal, Ian Wu, Viktor Zaverkin, Spase Petkoski, Daniel R. Schrider, Ilija Dukovski, Francesco Santini, Biljana Mitreska, Yong Jeong, Kyeongha Kwon, Young Min Sim, Dragana Manasova, Arthur Porto, Biljana Mojsoska, and 39 others. 2026a. On the limits and opportunities of AI reviewers: Reviewing the reviews of Nature-family papers with 45 expert scientists. Preprint, arXiv:2605.20668.

Kim et al. (2026b) Sunnie S. Y. Kim, Wesley Hanwen Deng, Jennifer Wortman Vaughan, Buxin Su, Weijie Su, Alekh Agarwal, Sharon Li, Martin Jaggi, Daniel G. Goldstein, Nihar B. Shah, and Miroslav Dudík. 2026b. Use and effects of LLMs in peer review: A randomized experiment and survey at ICML 2026. Preprint, arXiv:2609.19420.

Li et al. (2026) Chenjun Li, Cheng Wan, Haomiao Chen, and Johannes C. Paetzold. 2026. MELD: Multi-task equilibrated learning detector for AI-generated text. Preprint, arXiv:2605.06903.

Liang et al. (2024a) Weixin Liang, Zachary Izzo, Yaohui Zhang, Haley Lepp, Hancheng Cao, Xuandong Zhao, Lingjiao Chen, Haotian Ye, Sheng Liu, Zhi Huang, Daniel A. McFarland, and James Y. Zou. 2024a. Monitoring AI-modified content at scale: A case study on the impact of ChatGPT on AI conference peer reviews. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 29575–29620. PMLR.

Liang et al. (2024b) Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Yi Ding, Xinyu Yang, Kailas Vodrahalli, Siyu He, Daniel Scott Smith, Yian Yin, Daniel A. McFarland, and James Zou. 2024b. Can large language models provide useful feedback on research papers? A large-scale empirical analysis. NEJM AI, 1(8):AIoa2400196.

Ou et al. (2025) Jiefu Ou, William Gantt Walden, Kate Sanders, Zhengping Jiang, Kaiser Sun, Jeffrey Cheng, William Jurayj, Miriam Wanner, Shaobo Liang, Candice Morgan, Seunghoon Han, Weiqi Wang, Chandler May, Hannah Recknor, Daniel Khashabi, and Benjamin Van Durme. 2025. CLAIMCHECK: How grounded are LLM critiques of scientific papers? In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 21712–21735, Suzhou, China. Association for Computational Linguistics.

Padmakumar and He (2024) Vishakh Padmakumar and He He. 2024. Does writing with language models reduce content diversity? In The Twelfth International Conference on Learning Representations.

Russo et al. (2025) Giuseppe Russo, Manoel Horta Ribeiro, Tim R. Davidson, Veniamin Veselovsky, and Robert West. 2025. The AI review lottery: Widespread AI-assisted peer reviews boost paper scores and acceptance rates. Proceedings of the ACM on Human-Computer Interaction, 9(7):1–28.

Saha et al. (2026) Rounak Saha, Gurusha Juneja, Dayita Chaudhuri, Naveeja Sajeevan, Nihar B. Shah, and Danish Pruthi. 2026. Policies permitting LLM use for polishing peer reviews are currently not enforceable. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research, pages 106929–106954. PMLR.

Simpson (1960) George Gaylord Simpson. 1960. Notes on the measurement of faunal resemblance. American Journal of Science, 258-A:300–311.

Sourati et al. (2026) Zhivar Sourati, Alireza S. Ziabari, and Morteza Dehghani. 2026. The homogenizing effect of large language models on human expression and thought. Trends in Cognitive Sciences, 30(9):805–816.

Tolstykh et al. (2026a) Irina Tolstykh, Aleksandra Tsybina, Sergey Yakubson, Aleksandr Gordeev, Vladimir Dokholyan, and Maksim Kuprashevich. 2026a. GigaCheck: Detecting LLM-generated content via object-centric span localization. In Findings of the Association for Computational Linguistics: ACL 2026, pages 4349–4364, San Diego, California, United States. Association for Computational Linguistics.

Tolstykh et al. (2026b) Irina Tolstykh, Aleksandra Tsybina, Sergey Yakubson, and Maksim Kuprashevich. 2026b. LLMTrace: A corpus for classification and fine-grained localization of AI-written text. In Findings of the Association for Computational Linguistics: EMNLP 2026, pages 41585–41622, Budapest, Hungary. Association for Computational Linguistics.

ulab-ai (2025) ulab-ai. 2025. ResearchArcade-openreview-reviews. Hugging Face dataset. https://huggingface.co/datasets/ulab-ai/ResearchArcade-openreview-reviews.

Wu et al. (2025) Fan Wu, Emily Black, and Varun Chandrasekaran. 2025. Generative monoculture in large language models. In The Thirteenth International Conference on Learning Representations.

Wu et al. (2026) Wenqing Wu, Chengzhi Zhang, Yi Zhao, and Tong Bao. 2026. Impact of large language models on peer review opinions from a fine-grained perspective: Evidence from top conference proceedings in AI. Scientometrics, 131(5):3547–3584.

Xu et al. (2025) Jingjun Xu, Chongshan Lin, Haofei Yu, Tao Feng, and Jiaxuan You. 2025. ResearchArcade: Graph interface for academic tasks. Preprint, arXiv:2511.22036.

Yu et al. (2026) Sungduk Yu, Man Luo, Avinash Madasu, Vasudev Lal, and Phillip Howard. 2026. Is your paper being reviewed by an LLM? Benchmarking AI text detection in peer review. In The Fourteenth International Conference on Learning Representations.

Appendix A Data

Sampling.

OpenReview and journal collection took place on 27–29 October 2026, with more journal papers on 6 November 2026. ICLR 2027 reviews come from 15 November 2026, eight days after they became public and before decisions. ICLR 2026 reviews come from a snapshot of 21 November 2025 (Heineman, 2025), ten days after the reviews became public. After that date, the organizers reset all reviews to their state at the start of the discussion phase (ICLR 2026 Program Chairs, 2025), so the public record holds no final version of these reviews. The journal year 2026 covers January–October. BMC Medicine ends in 2025. We sampled all ICLR (2020–2027) and NeurIPS (2021–2025) papers with at least three reviews, all ICML accepted papers in 2025 and 2026, and up to 2,000 papers per journal source-year with at least two first-round reviews. SciPost reviews come from its API. We obtain F1000Research, Wellcome Open Research, Open Research Europe and PLOS reviews from Europe PMC, BMC Medicine reviews from JerMa88 (2025), and Copernicus/EGU reviews from journal discussion pages. We retain first-round referee reviews, exclude author replies and editor notes, and cap papers at five reviews. Withdrawn and desk-rejected papers do not enter the main sample before 2026. The ICLR samples of 2026 and 2027 keep them, so that both years hold all papers with three reviews. Table 5 gives each conference source-year and each journal source in two periods.

Flagged (%)

Source Year Papers Reviews Claims per review Overlap (%) MELD GigaCheck

ICLR 2020 2,127 6,543 7.1 10.5 ±\pm 0.3 0.0 0.0

2021 2,594 10,019 7.7 9.2 ±\pm 0.2 0.0 0.1

2022 2,617 10,188 7.6 10.0 ±\pm 0.2 0.0 0.3

2023 3,789 14,271 7.1 10.8 ±\pm 0.2 0.1 0.1

2024 5,690 21,934 6.9 8.9 ±\pm 0.1 6.1 5.8

2025 8,629 34,689 7.2 9.1 ±\pm 0.1 17.8 15.9

2026 19,461 75,674 7.2 12.0 ±\pm 0.1 44.6 16.5

2027 40,559 125,850 7.4 21.3 ±\pm 0.1 62.7 16.8

NeurIPS 2021 2,768 10,696 7.3 8.3 ±\pm 0.2 0.0 0.1

2022 2,820 10,313 7.0 8.1 ±\pm 0.2 0.0 0.1

2023 3,394 14,937 6.8 8.2 ±\pm 0.1 3.7 2.3

2024 4,235 16,596 6.6 8.0 ±\pm 0.1 11.4 5.4

2025 5,529 22,324 6.7 9.1 ±\pm 0.1 18.6 9.4

ICML 2025 3,256 12,449 7.0 8.4 ±\pm 0.1 35.5 16.6

2026 6,332 24,355 6.5 11.6 ±\pm 0.1 51.5 30.2

SciPost 2019–23 1,586 3,538 7.3 10.3 ±\pm 0.5 1.0 0.4

2024–26 1,376 3,075 7.8 10.9 ±\pm 0.5 11.5 3.1

F1000 2019–23 3,864 8,934 7.4 10.7 ±\pm 0.3 2.6 2.5

2024–26 3,247 8,049 6.6 12.6 ±\pm 0.4 29.3 22.5

PLOS 2019–23 9,259 21,684 10.0 8.8 ±\pm 0.2 1.6 1.0

2024–26 5,101 12,400 10.0 9.6 ±\pm 0.2 31.4 22.9

BMC-Med 2019–23 1,802 4,610 10.4 7.8 ±\pm 0.3 1.2 0.8

2024–25 708 1,804 9.9 7.6 ±\pm 0.4 16.9 12.9

Copernicus 2019–23 10,000 21,872 20.2 11.7 ±\pm 0.1 0.7 0.4

2024–26 6,000 13,005 19.9 12.3 ±\pm 0.2 21.3 6.9

Table 5: Each conference year and each journal source in two periods. Overlap is the mean over papers with its SE. Flagged: share of reviews that each detector flags at its own threshold (Appendix D).

Source differences.

ICML changed its form between 2025 and 2026 and also changed its LLM policy (ICML 2025 Program Chairs, 2025; ICML 2026 Program Chairs, 2026a; ICML 2026 Program Chairs, 2026b). ICML 2026 assigned each reviewer one of two policies (Kim et al., 2026b), but public records do not reveal the policy attached to each review. Open journals can also expose later reviewers to earlier reviews. These differences limit comparisons of levels across venues.

ICLR review forms.

The 2023 ICLR form raised overlap in that year. ICLR changed the free-text fields of its form in 2022, 2023, 2024 and 2027 (Table 6). Only the 2023 form had a separate field on clarity, quality, novelty and reproducibility, which asks each reviewer about the same topics. In 2023, 50% of the shared problems of a paper include a claim on clarity, novelty or reproducibility, against 42% in 2022 (last column of Table 6).

Two tests measure the effect of this field. First, we delete the clarity field from each 2023 review and run both passes again. Overlap changes by −1.1-1.1 points. Deleting any field can change overlap, so as a control we delete the questions field from each 2024 review. Overlap then changes by only −0.2-0.2 points. Second, we keep each review whole and remove from the existing groups each claim that Luna labels as clarity, novelty or reproducibility. Overlap is then 8.7% in 2022, 8.4% in 2023 and 8.2% in 2024, so 2023 no longer stands above the years next to it.

Overlap (%)

Year Text fields Papers Whole reviews Field deleted Categories dropped Shared problems in categories (%)

2020 review 2,127 10.5 – 9.0 42.8

2021 review 2,594 9.2 – 7.9 42.7

2022 summary; main review; review summary 2,617 10.0 – 8.7 42.0

2023 summary; strengths and weaknesses; clarity, quality, novelty and reproducibility; review summary 3,789 10.8 9.7 8.4 49.7

2024 summary; strengths; weaknesses; questions 5,690 8.9 8.7 8.2 39.0

2025 summary; strengths; weaknesses; questions 8,629 9.1 – 8.7 35.5

2026 summary; strengths; weaknesses; questions 19,461 12.0 – 11.9 28.1

2027 summary; critical strengths; critical weaknesses; questions 40,559 21.3 – 21.9 21.7

Table 6: Only the 2023 ICLR form had a separate field on clarity, quality, novelty and reproducibility (bold). Field deleted: we delete the clarity field in 2023 and, as a control, the questions field in 2024, and run both passes again. Categories dropped: we remove the claims on clarity, novelty and reproducibility from the existing groups. Shared problems in categories: share of the shared problems of a paper that include at least one claim on clarity, novelty or reproducibility. All values are means over papers.

Models and release.

Both passes use GPT-6 Luna through OpenRouter with strict JSON-schema output and high reasoning effort. Journal prompts use the generic rather than the ML paper type. Each call gets four attempts, and a schema violation counts as a failure. No review failed all four attempts. Saved requests record API model names and settings. We release the code,11 1 https://github.com/IlyaGusev/twicetold all collected reviews,22 2 https://huggingface.co/datasets/IlyaGusev/twicetold paper metadata, extracted claims, problem groups and per-review MELD scores. Journal reviews retain their CC BY or CC0 terms. All texts were public before collection.

Appendix B Extraction and Grouping Checks

Human labels.

Luna finds a human item (Section 3.2) when the quote of one of its claims shares five consecutive words with the item, or all words of the shorter text. In the Nature-family set, 30.0% of the items receive more than one claim, and Luna lists 3.0 more claims per review than the set lists items. CLAIMCHECK marks only weaknesses tied to a paper claim, so we give only recall for that set. No public set carries human labels of same-problem groups across reviews, so pass 2 has no human check.

Council sample.

We fixed the prompts and then drew 510 same-paper and 90 cross-paper panels, stratified by source (ICLR 2018–2027, NeurIPS 2024–2025, ICML 2025–2026 and five journals 2024–2026). The ICLR panels of 2018 and 2019 are outside the analysis years and serve only this check. From 2024 on, we also stratified by detector flags: panels with no flagged review and panels with at least two flagged and one unflagged review. Council models run through OpenRouter at medium reasoning effort. For one review, the span F1 between two extractors is the F1 between the sets of word positions that their quotes cover. This score gives partial credit to partial overlaps and does not depend on how finely either side splits claims (Table 7). The sample is stratified, so values on it are not estimates for the corpus.

Claims Claims Span F1 Span F1

Panels Detector stratum nn Luna council Luna council

Same All 510 7.6 7.4 78.5 80.5

Same plain 311 7.7 7.5 78.2 80.4

Same flagged 199 7.4 7.3 79.0 80.7

Cross All 90 8.4 8.3 78.7 81.0

Table 7: Luna’s quotes agree with those of a council member almost as closely as the quotes of two members agree, in each detector stratum. Claims are per review. Span F1 compares the quoted words of Luna and one member, or of two members.

Claim grouping.

The label of a claim pair is the majority vote of the three members on whether the two claims belong to one group. We keep a pair only when the outputs of all three members contain both claim ids exactly once. This rule keeps 141,426 of 141,700 cross-review claim pairs (99.8%). Luna agrees with 1,148 majority links, misses 316 and adds 226 (Table 8). This check covers 507 of the 510 same-paper panels, because the other 3 have claims in at most one review and give no claim pair. Unlabeled pairs do not define complete reference groups, so this check does not validate full-panel overlap levels.

Grouping error by stratum.

Per 100 majority links, Luna finds 87 links in panels with no flagged review and 101 in the flagged stratum (Table 8). This difference can change comparisons between papers. Within the panels of the flagged stratum, Luna finds 100 links per 100 majority links between two unflagged reviews and 101 between two flagged reviews, so we find no difference in this error between the pair types of one paper. This check rests on 70 majority links between two unflagged reviews. With a council member or the majority as grouper, the overlap difference between flagged and plain panels falls from 5.9 points to 3.9–5.1 points, and the within-paper gap changes little (Table 2).

Panels Detector stratum nn Pairs Precision Recall F1 Links per 100

Same All 507 141,426 83.6 78.4 80.9 94

Same plain 308 81,465 83.7 72.5 77.7 87

Same flagged 199 59,961 83.4 84.2 83.8 101

Table 8: Luna’s grouping agrees with the majority vote of the council on same-paper panels, but misses more links in panels with no flagged review. Pairs: claim pairs from different reviews of one panel with a vote from all three members. Precision, recall and F1 take the majority vote as truth. Links per 100: Luna’s links per 100 majority links.

Full pipelines.

In a full pipeline, one model extracts and groups its own claims (Table 3). On one panel, the overlap of Luna’s pipeline differs from that of a member by 5.5 points on average, and the overlaps of two members differ by 5.5 points. The 95% interval of the within-paper flagged-pair gap excludes zero in 3 of four pipelines, and that of the ICLR step from 2026 to 2027 in 4 of four. In corpus panels that hold one review from each of four ICLR papers of one year, overlap is 0.2% in 2026 and 0.1% in 2027, so the grouper shows no drift toward merging unrelated claims; this check cannot detect drift in same-paper claims.

Appendix C Gap Estimators and Overlap Measures

Flagged-pair gap.

We compare within papers, because a plain difference between all two-flagged pairs and all unflagged pairs mixes the flags with differences between papers. For a paper pp, let o¯ptwo\bar{o}_{p}^{\,\mathrm{two}} and o¯pnone\bar{o}_{p}^{\,\mathrm{none}} be the mean overlap of its pairs with two flagged reviews and with no flagged review. The flagged-pair gap is

Δ^=1|P|​∑p∈P(o¯ptwo−o¯pnone),\hat{\Delta}=\frac{1}{|P|}\sum_{p\in P}\left(\bar{o}_{p}^{\,\mathrm{two}}-\bar{o}_{p}^{\,\mathrm{none}}\right), (2)

where PP is the set of papers with both pair types. Such a paper has at least two flagged and two unflagged reviews.

Fixed-effects estimate.

We also fit

op​k=αp+β1​onep​k+β2​twop​k+εp​k,o_{pk}=\alpha_{p}+\beta_{1}\,\mathrm{one}_{pk}+\beta_{2}\,\mathrm{two}_{pk}+\varepsilon_{pk}, (3)

where op​ko_{pk} is the overlap of pair kk of paper pp, αp\alpha_{p} is a paper fixed effect, and onep​k\mathrm{one}_{pk} and twop​k\mathrm{two}_{pk} mark a pair with one and with two flagged reviews. The estimate is β^2\hat{\beta}_{2}. This fit also uses the three-review papers with mixed flags, so the main text uses it for the journals, where few papers have four reviews. For the pooled journals, this fit gives 5.4 points from 2,228 papers with mixed flags, and Equation 2 gives 5.0 points from 230 papers. At ICLR, this fit gives 2.7 and 7.7 points for the two periods without controls and 3.3 and 7.2 with log length and log claim-count controls, against 3.0 and 4.5 from Equation 2. It also uses papers without a two-flagged or an unflagged pair and weighs pairs, not papers, equally. Its SEs cluster by paper.

Overlap measures.

Any ratio of shared problems to set size depends on set size, and the raw shared count grows with review length. Table 9 therefore repeats each main claim with seven measures and holds set size fixed in two of them.

•

Overlap coef.: 100​o100\,o, the main measure.

•

Jaccard: 100​|GA∩GB|/|GA∪GB|100\,|G_{A}\cap G_{B}|/|G_{A}\cup G_{B}|.

•

Dice: 200​|GA∩GB|/(|GA|+|GB|)200\,|G_{A}\cap G_{B}|/(|G_{A}|+|G_{B}|).

•

Shared: the count |GA∩GB||G_{A}\cap G_{B}|.

•

Shared, linear sizes: the shared count with the smaller and the larger set size as controls.

•

Shared, same sizes: the shared count with one indicator per pair of set sizes, so a four-and-ten pair is compared only with other four-and-ten pairs.

•

New problems: for changes over time, the problems that a third review adds to two others. For within-paper gaps, the problems of one review that no other review of the paper raises. This measure counts novelty, so the claims predict a negative sign.

Claim Overlap coef. Jaccard Dice Shared Shared, linear sizes Shared, same sizes New problems

points points points per pair per pair per pair per review

Change over time

ICLR, 2026 to 2027 +9.3 +3.9 +6.2 +0.49 +0.49 +0.49 −-0.66

ICLR, 2025 to 2026 +2.9 +1.3 +2.3 +0.16 +0.15 +0.15 −-0.26

NeurIPS, 2024 to 2025 +1.1 +0.6 +1.0 +0.07 +0.05 +0.05 −-0.00†

ICML, 2025 to 2026 +3.2 +1.6 +2.7 +0.15 +0.16 +0.16 −-0.66

F1000, 2019–23 to 2026 +5.8 +3.1 +4.1 +0.47 +0.54 +0.48 −-0.78

PLOS, 2019–23 to 2026 +2.5 +1.1 +1.9 +0.28 +0.26 +0.27 −-0.18†

Within-paper gap against unflagged pairs or reviews

ICLR 2024–25, two flagged +2.6 +1.4 +2.1 +0.03† +0.13 +0.12 −-0.04

ICLR 2026–27, two flagged +7.7 +3.4 +5.4 +0.44 +0.41 +0.42 −-0.29

NeurIPS, two flagged +4.4 +2.1 +3.4 +0.15 +0.21 +0.21 −-0.13

ICML, two flagged +4.5 +2.0 +3.4 +0.21 +0.22 +0.22 −-0.18

Journals, two flagged +5.4 +2.2 +3.4 +0.32 +0.35 +0.34 −-0.27

Table 9: Every main claim keeps its sign under each pair measure. The claims predict a positive sign for pair measures and a negative sign for new problems. † marks an estimate that does not differ from zero in that direction at the 95% level. A change over time is the last year minus the base year, or minus the 2019–2023 mean for journals. A within-paper gap compares two flagged with two unflagged reviews of one paper, in a regression with paper fixed effects.

Results.

Every

正文以来源语言显示。

参考来源

← 返回每日AI报告