Answer Reviewer 2 before Reviewer 2 exists.
Prebuttal is a pre-submission stress test for ML papers: three AI reviewers with different temperaments — running in fully independent contexts — plus an area chair, delivering a conference-format review report before the real Reviewers 1–3 see your manuscript.
Why not just paste the paper into a chatbot?
One conversation role-playing three reviewers collapses into one bland compromise opinion.
Three reviewers, zero shared context
Each reviewer is a separate call or process with its own persona and model — verdicts genuinely differ, and the disagreement (σ) is reported, always.
Real review forms
NeurIPS 1–6, ICLR 1–10, ICML 1–10 and ARR Soundness/Excitement/Overall —
with the official scale anchors, not invented ones. Another venue? Any form
and scale is one TOML file: prebuttal venues new cvpr.
Cell-accurate ingestion
LaTeX parsing, PDF table extraction with per-cell geometry, caption fallback — reviewers cite your “Table 3, column 4”, not a guess.
Honest by construction
Every review is labeled with its backend and model; a calibration script measures scores against real OpenReview ratings (results not yet published); run history diffs weaknesses across revisions.
Sample runs
Full runs on public arXiv papers, compared with their real OpenReview outcomes. Unofficial demos — not written or endorsed by the papers' authors.
Mamba — one paper, two rounds: the rejected v1 vs the revised v2
The same cross-model panel (claude sonnet · codex gpt-6-luna · gemini-3.8-flash, AC opus) reviewed both arXiv versions of Mamba, three seeds each. The score did not move (mean 7.22 → 7.33, inside the noise) and, matched by meaning, every concern in the v1 revision plan was raised again against v2 — including the missing error bars the real ICLR reviewers hammered on. The tool says “no change” when it sees none.
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Two reviewers at borderline accept, the detail reviewer at accept (4 / 4 / 5), and an AC estimate of 5. The revision plan goes after what a skeptic would: no matched PPO baseline for dialogue, no uncertainty on any win rate, an unmeasured efficiency claim. Positive, as the real outcome was.
GlobalGate: A Universal Attention Mechanism That Solves Long-Context Reasoning
What a weak submission gets. A fictional paper written for this demo with deliberate flaws — baseline trained 1 epoch vs. 5, hyperparameters tuned on the test set, one seed, no competing methods, "universal" claims from one 2k-token task. All three reviewers caught every flaw and cited the paper's own lines.
Before you submit: the desk-reject pre-flight
No model, no cost, a few seconds: anonymity leaks, the venue's page
limit and required sections, and leftovers like ?? or hidden text.
# real output on the bundled demo paper $ prebuttal check paper.pdf --venue icml prebuttal check — paper.pdf · ICML rules (8 pages, double-blind, requires Impact Statement) [x] format no "Impact Statement" section — the venue requires one [x] leftovers unresolved references (??) on page 2 [ok] PDF metadata has no author [ok] no self-revealing links or citations [ok] main text 2 of 8 pages [ok] no hidden text 2 problems that can get the paper desk-rejected.
Anonymity
Author names in the PDF metadata, e-mails on page 1, a camera-ready or
[preprint] build, a visible acknowledgements section, links to
your own repository, “our previous work [12]”.
Format and leftovers
Main-text pages over the limit (two-column layouts too), a missing NeurIPS
checklist / ICML impact statement / ARR limitations section, ??,
[?], TODOs. LaTeX projects are checked without compiling.
Exit code 6 drops it into CI.
The revision loop
Every run ends with a prioritised revision plan; every paper is a project whose runs line up as revision history — and two runs can be diffed.
# abridged output of the real runs $ prebuttal run 2312.00752v1 --project mamba-revision --note "v1 — rejected at ICLR 2024" R1 claude/sonnet 8 · R2 codex/gpt-6-luna 6 · R3 gemini-3.8-flash FAILED (HTTP 429) the report uses the 2 reviews that finished — `prebuttal retry mamba-revision#11` $ prebuttal retry mamba-revision#11 # only R3 and the AC run again R3 gemini-3.8-flash 8 Predicted: 7.33/10 (σ 0.94) → saved as mamba-revision#17 $ prebuttal run 2312.00752v2 --note "v2 — revised, accepted at COLM 2024" $ prebuttal retry mamba-revision#12 # R3 hit the same quota; completed the same way Predicted: 7.33/10 (σ 0.94) → saved as mamba-revision#18 $ prebuttal diff --llm mamba-revision#17 mamba-revision#18 panel 7.33/10 → 7.33/10 (Δ +0) still raised: [must] Headline claims exceed the tested regime … still raised: [should] No seeds, standard deviations or confidence intervals … new in B: the custom fused CUDA kernel's portability 0 of 8 plan items were not re-detected in B.
A revision plan, not just a verdict
The area chair merges what the reviewers raised into a short checklist —
must / should / could, the issue with its table or section, the
concrete fix, who raised it. prebuttal plan prints it; the next
prebuttal diff checks it against the re-run — by meaning with
--llm.
Projects & run history
Each paper gets a project found from its source — the same PDF, LaTeX
folder or arXiv id always lands in the same project, even after a title
change. Runs are numbered (mamba#3) and carry your
--note "what changed".
A local overview
prebuttal projects --open writes one offline HTML page with every
project, score trends and links to each report — no server, no network,
nothing leaves your machine. The archive itself is user-only
(0700).
Reports that never overwrite
Every run writes its own folder —
prebuttal-reports/<project>/<date>_<venue>/ —
plus latest.md / latest.html, with a built-in
.gitignore so reviews of unpublished work stay out of version
control.
Bring your own backend
Auto-detected at startup; every run records who reviewed what.
| Backend | Cost | Notes |
|---|---|---|
agent-claude / agent-codex | subscription, zero API key | cross-model reviews when both exist |
api-gemini | free tier | the cheapest “first key” path |
api-openrouter | :free ids cost nothing | one key, hundreds of models; live prices in prebuttal backends api-openrouter |
api-openai / api-anthropic | BYO key | mid-tier models by default |
ollama | free, local | unpublished papers never leave your machine |
relay | free | you, your agent’s subagents, or any system answers by hand |
ollama or
relay.