Indication No. 054 Generated cover image

PaperOrchestra: AI agents turn research notes into a manuscript

PaperOrchestra automates the path from research notes to a complete manuscript and clearly outperforms other AI writing systems. Its impressive evaluation results raise questions about the established publication cycle.

Google researchers Yiwen Song, Yale Song, Tomas Pfister and Jinsung Yoon introduce PaperOrchestra, a system that transforms existing idea summaries and experimental logs into complete scientific manuscripts. The work is listed as a COLM 2026 conference publication; an earlier arXiv version is also freely available. [1][2][3]

An orchestrated writing process

Specialized AI agents handle five stages: outlining, generating diagrams and data visualizations, researching and synthesizing literature, writing sections, and refining the manuscript using simulated peer review. Literature and visualization work can run in parallel. The outputs are LaTeX source files and a typeset PDF. Retrieved publications are checked for existence through Semantic Scholar. This check does not guarantee that every cited claim is substantively supported. [3]

The distinctive step is treating writing as a standalone workflow: Materials can come from human research rather than necessarily from an automated experimental pipeline. Experiments and research results must already be supplied as inputs. PaperOrchestra therefore demonstrates automated manuscript production, rather than the complete independent generation and validation of new research. [3]

The impressive overall evaluation result

The benchmark uses 200 published papers, 100 each from CVPR and ICLR 2025. Idea summaries and experimental logs were reverse-engineered from these completed papers. Alongside automated assessments, 11 AI researchers evaluated a sample of 40 papers. [3]

In human pairwise comparisons of overall quality, PaperOrchestra won 57 percent against the single-agent system, which won 19 percent, with 24 percent ties. Against AI Scientist-v2, PaperOrchestra led 50 to 36 percent, with 14 percent ties. These are win-rate margins of 38 and 14 percentage points, respectively. They represent neither a universal 14–38 percent increase in scientific quality nor conference acceptance rates. [2][3]

This is a remarkable result for an automated writing workflow. Yet comparison with human originals remains clear: For overall quality, human evaluators preferred the original in 81 percent of comparisons and PaperOrchestra in 5 percent, with 14 percent ties. These results do not establish general parity with human-written papers. [2]

Limits of the evidence

The study concerns AI and machine-learning papers and artificially reconstructed starting materials. Despite mitigation measures, model familiarity with the published originals cannot be fully excluded. The central baseline comparisons also supplied original figures; autonomous visualization was assessed separately. Generated figures can contain errors, and automated reviewers show limited agreement with human judgements, particularly for literature reviews. The authors explicitly retain human researchers’ responsibility for factual accuracy, originality and validity. [3]

A question for the scientific publication cycle

Our closing question concerns the purpose of the process: Does the established scientific publication cycle still make sense when AI can turn existing results into manuscripts, simulate their review and revise them—and how should that cycle change to reward genuinely new, verifiable knowledge rather than primarily persuasive presentation?

Sources

  1. [1] Google Research: PaperOrchestra: A Multi-Agent Framework for Automated AI Research Paper Writing (2026) (opens in a new tab)
  2. [2] Song, Song, Pfister & Yoon (2026): PaperOrchestra – COLM-Konferenzpublikation (PDF) (opens in a new tab)
  3. [3] Song et al. (2026): PaperOrchestra – frei zugänglicher Volltext der arXiv-Fassung vom 6. April 2026, CC BY 4.0 (opens in a new tab)

This indication by Martin Ebner is licensed under
CC BY 4.0CCBY

Comments 0

Comments are moderated. New comments appear after review by the editors.

No comments yet.

Write a comment

Published with the contribution.

Only used to tell you about the status and for questions from the editors. Never published and deleted 30 days after the decision.

Comments are reviewed before publication. No HTML; links are allowed.

Spam protection

To prevent spam we use Cloudflare Turnstile. Loading it connects your browser to Cloudflare. More in the privacy policy

Prefer not to load Cloudflare? Feel free to send your contribution by email to redaktion@postgenerativ.at.