PaperOrchestra Explained: Multi-Agent Research Writing With Verifiable Evidence
How Google's PaperOrchestra turns research materials into LaTeX, where its agents hand off artifacts, and how to validate citations, experiment claims and plots. Includes a runnable evidence-checking exercise and an original architecture diagram.
BY EMILY CARTER · FDEINTERVIEWS EDITORIAL · UPDATED NOVEMBER 15, 2026 · 4 MIN READ
PRACTICE THIS:Understand multi-agent orchestration ·Learn harness engineering ·Explore research papers for FDEs
PaperOrchestra is a multi-agent framework that turns ideas, experimental logs and document requirements into a LaTeX research manuscript. Its value for FDEs is the separation of responsibilities: build an outline, gather literature, produce figures, write sections and refine the result. Each handoff creates an opportunity to check evidence before a mistake spreads through the document.
The official repository is a research project, not an officially supported Google product. We inspected its CLI and implementation. We did not run the paid model pipeline. The local exercise below is our own evidence-validation example, which runs without API keys.
Begin with what the system actually receives
The PaperOrchestra paper studies manuscript generation from pre-writing materials. Supplying an experiment log is different from having the system establish that the experiment was valid. A polished paper cannot repair mislabeled test data or a measurement taken under the wrong configuration.
For a customer evaluation report, prepare a brief, versioned experiment outputs, the evaluation protocol and the audience's requirements. Keep unsuccessful runs. Give each result a stable identifier that the writer can cite internally.
This applies the same distinction as verification in agent harnesses: producing the requested artifact and proving its claims are separate responsibilities.
Follow the artifacts through the agents
The project coordinates outline, literature, plotting, section-writing and refinement work. Read the project overview alongside the diagram. The green checks below are recommended controls around those handoffs; they are not claims that the upstream project enforces every check.
An outline should identify the purpose and evidence requirements of each section. Literature work should deliver sources that support those requirements. A plotting worker should deliver a figure plus the data and code that produced it. The section writer combines those artifacts, and refinement should preserve their meaning while improving presentation.
This is a practical case of multi-agent orchestration. Splitting roles helps when the outputs have clear contracts. If each agent receives the entire conversation and returns unstructured advice, adding agents mainly creates more text to reconcile.
Inspect the CLI before spending on a run
The CLI source names the expected inputs. The defaults include idea_sparse.md and experimental_log.md; template and output paths are separate arguments. Model choices for writing, reflection, plotting and images are distinct settings.
You can inspect the interface without invoking any model:
git clone https://github.com/google-research/paper-orchestra.git
cd paper-orchestra
python3 paper_writing_cli.py --help
Before an actual run, follow the repository's installation instructions and review the configured providers for every enabled stage. The shipped defaults are not a procurement decision. Record the source revision, model identifiers, input hashes and run configuration together.
Use a dedicated workspace. Generated plotting code and document compilation belong in a sandbox with bounded execution time and restricted filesystem access. The agent sandboxing concept explains the difference between a convenient subprocess and an isolation boundary. Keep network access for literature lookup separate from the worker executing generated code.
Make numerical claims traceable
Consider a report that says a candidate answered 86% of cases correctly. The important questions are which cases, which model configuration and whether the denominator includes failures. Store those facts before asking an agent to write the sentence.
This small Python example validates a proposed claim against a synthetic run record. It is an application control, not a PaperOrchestra API.
from decimal import Decimal
runs = {
"candidate-v2": {
"dataset": "support-v3-test",
"correct": 43,
"total": 50,
}
}
def validate_claim(claim):
run = runs[claim["run_id"]]
if claim["dataset"] != run["dataset"]:
raise ValueError("Dataset mismatch")
if not 0 <= run["correct"] <= run["total"] or run["total"] == 0:
raise ValueError("Invalid counts")
actual = Decimal(run["correct"]) / Decimal(run["total"])
if Decimal(claim["value"]) != actual:
raise ValueError("Claim does not match the recorded result")
return f'{run["correct"]}/{run["total"]} = {actual:.0%}'
claim = {
"run_id": "candidate-v2",
"dataset": "support-v3-test",
"value": "0.86",
}
print(validate_claim(claim))
try:
validate_claim({**claim, "value": "0.90"})
except ValueError as error:
print(error)
The expected results are 43/50 = 86% and a rejection of the altered claim. This only checks consistency with the ledger. It does not establish that the dataset was representative or that the grader was correct. Those require evaluation design and an independent test protocol.
For a plotted comparison, also validate units, aggregation and sample counts. Give the plotting worker the measured series, not a prose instruction to draw an improvement. A figure should be reproducible from its data even if the writing model is unavailable.
Verify citations in two passes
First check identity: does the cited paper exist, and do its title, authors and identifier match the bibliography? Then check support: does the cited passage justify the sentence, including its qualifications?
A citation can pass identity checks while failing support. A paper about performance on one benchmark cannot establish a general production reliability claim. Keep the relevant passage beside the claim during review, and mark unresolved claims explicitly.
The Keenable search integration shows how search and fetching can populate that evidence record. Our deep research review-layer analysis covers what happens when a reviewer sees incomplete sources or fails open.
What the benchmark can and cannot establish
PaperWritingBench contains 200 paper-writing tasks, with source materials reconstructed from existing papers. The paper's benchmark description is useful for comparing writing systems under a common setup. It is not a prospective trial of original discoveries or a guarantee of publication acceptance.
Reconstruction also changes the starting point: source papers already contain a coherent research story. Messy customer logs may lack that structure. Evaluate on your own document type, including contradictory results, missing evidence and a result that should remain inconclusive.
Adapt the pattern to customer work
Three plausible applications are an evaluation report, a migration decision memo and a reproducibility appendix. In each case, define the evidence contract before introducing specialist agents. A migration memo needs verified compatibility constraints; an evaluation report needs a stable dataset and metrics; an appendix needs commands and versioned artifacts.
Start with one document and a human reviewer. Measure correction time and unsupported claims, not just writing speed. The harness engineering course develops these execution boundaries, and the papers library gives you research to practice reviewing. The useful outcome is a document whose claims can be traced and checked after the agents have stopped running.
Turn it into offers. Work the real questions and concepts this maps to:
FAQ
PaperOrchestra is a Google Research open-source framework for turning pre-writing materials into research manuscripts. Specialized agents handle outlining, literature review, plotting, section writing and refinement. The repository explicitly states that it is not an officially supported Google product.
Its stated task starts from ideas and experimental logs and produces a manuscript. That should not be confused with independently designing, executing and validating the underlying experiments. A generated results section still needs to match the supplied measurements.
Check numerical claims against versioned experiment records, citation metadata against primary sources, and source support against the exact claim. Reproduce plots from the recorded data and review the final document. Successful compilation and favorable model reviews do not establish scientific correctness.
The separation between evidence gathering, structured outlining, writing and review can also inform evaluation reports or technical proposals. Those are adaptations of the design, not features independently validated by the PaperOrchestra benchmark. Keep the required evidence and approval rules specific to the document.
Discussion (3)
An evaluator can reward fluent explanations while missing an unsupported result. The experiment ledger needs a deterministic check before prose quality gets a score.
Plotting is a code-execution boundary. Give that worker the data it needs and a disposable output directory, not the author's workstation credentials.
A useful handoff names what is missing. An empty citation field should lead to an unresolved claim, not a plausible-looking bibliography entry.
