Mamba, two rounds: the rejected v1 vs the revised v2
ICLR review form · one fixed cross-model panel (claude sonnet · codex gpt-6-luna · gemini-3.8-flash, area chair claude opus) · same personas and settings — full v1 report → · full v2 report →
The two panels (seed 42)
| Role | Model | v1 · rejected | v2 · revised | real ICLR 2024 |
|---|---|---|---|---|
| R1 · senior_kind | claude sonnet | 8 | 8 | 8 |
| R2 · adversarial | codex (gpt-6-luna) | 6 | 6 | 3 (the dissenter) |
| R3 · detail_obsessive | gemini-3.8-flash | 8 | 8 | 6 |
| Panel mean | 7.33 (σ 0.94) | 7.33 (σ 0.94) | 6.25 → reject | |
| AC estimate | claude opus | 8 | 8 |
What happened between rounds: almost nothing
The same concerns, raised again
Matched by meaning (prebuttal diff --llm), all 8 items of the v1
revision plan were raised again against v2: headline claims beyond the tested
scale (scaling laws stop near 1.3B, zero-shot at 2.8B), no seeds or error bars in
any table, a non-monotonic Table 13, a missing Transformer++ row in Table 3,
efficiency numbers without operating context. The one new v2 item is the
portability of the custom fused CUDA kernel. To this panel, the revision did not
address what it cared about.
The score doesn't move either — and that is measured, not guessed
Each version was reviewed three times (seeds 42–44). v1: 7.33, 7.33, 7.00 (mean 7.22). v2: 7.33, 7.33, 7.33 (mean 7.33). The whole difference is one point from one reviewer in one run — inside the noise. The senior and detail reviewers gave 8 to both versions in every run: this panel does not separate the two versions, and an earlier version of this page that read a 7.0 → 7.3 “improvement” from single runs was reading noise.
A claim we withdrew
That earlier page also said a “Mamba-3B vs 2.8B” naming inconsistency was
“gone in v2, because the revision fixed it”. It is not: the seed-42 v2 review
raises it again. The old claim came from word-overlap matching, which on this
pair calls 5 of the 8 plan items resolved — by meaning, none were. That is why
diff --llm exists.
The area chair still checks its panel
In both rounds the AC (claude opus) sided with the two 8s but kept the adversarial reviewer's points on scoping the claims. In v2 it disputed part of R2's case: the decisive Table 3 comparisons share tokenizer, data and token budget, and R2's reading of Table 15 conflated a memory benchmark with the KV-cache argument.