Prebuttal

Mamba: Linear-Time Sequence Modeling with Selective State Spaces

ICLR · simulated review · 2026-10-27T07:35:43+00:00 · seed 42
backends: R1: agent-claude · R2: agent-codex · R3: api-gemini
Unofficial demo: an AI-simulated review of arXiv:2312.00752v1 by Albert Gu and Tri Dao, licensed CC BY 4.0. This is arXiv v1 (December 2023) — the version rejected at ICLR 2024. Summaries and short quotes below were generated by AI models from the paper and may contain errors; generated only to show what Prebuttal outputs. Not written, reviewed or endorsed by the authors, and not an assessment of the work's merit. Authors who want this page removed: open an issue.
7.33/10
estimated · σ 0.94 · range 6–8 · n=3
Three independent AI reviewers, isolated contexts, plus an area chair. This is a pre-submission stress test, not a prediction.
R1 senior_kind 8/10
agent-claude · sonnet · conf 4

The paper argues that a key weakness of subquadratic sequence models, notably LTI structured SSMs, is the lack of content-based selection, and proposes making the SSM parameters (Δ, B, C) functions of the input (Section 3.2, 'S6'). Because this breaks the convolutional form, the authors give a hardware-aware scan using kernel fusion, parallel scan and recomputation (Section 3.3.2), and integrate it into a homogeneous gated block without attention or MLP (Mamba, Section 3.4). Experiments cover synthetic tasks…

Top weaknesses
  • Scale of evidence for the headline claim 'matches Transformers twice its size' and 'first to match Transformer++': scaling laws stop at ~1.3B and ~26B tokens at the largest point (Table 12), and the main Table 3…
  • Baseline fairness in Table 3 is uneven: the Pythia models are fixed off-the-shelf checkpoints, while Mamba is trained with tuned recipes, and Mamba-2.8B is compared to GPT-J-6B/OPT-6.7B (different data/tokenizers) for…
  • Weak or missing hybrid/attention comparisons at scale: Section 4.6.1 defers interleaving with MHA/MLP to Appendix E.2.2, and the SC09 ablation (Table 5) shows MHA+MLP in the center is worst, but attention is used in a…
R2 adversarial 6/10
agent-codex · gpt-6-luna · conf 4

This paper introduces input dependent selection for structured state space models, making their recurrent dynamics depend on the current token, and presents a fused parallel scan to compute the resulting model efficiently. The authors combine this layer with a simplified block architecture, Mamba, and evaluate it on synthetic tasks, language, DNA, and audio. The results show large gains over earlier attention free models and strong performance against several Transformer baselines, alongside faster autoregressive…

Top weaknesses
  • The broad foundation model claim is ahead of the language evidence. Section 4.2 scales only to about 1.3B parameters, and Table 3’s largest Mamba model is 2.8B; the paper itself acknowledges in Section 5 that it has…
  • The strongest language baseline is not fully covered in downstream evaluation. Section 4.2.1 compares scaling against Transformer++, but Table 3’s zero shot comparison omits it; the statement that Mamba is best in…
  • There is no reported seed variation, confidence interval, or significance analysis for the main comparisons. This matters especially for Table 3’s smaller margins, such as the 0.5 point average advantage of Mamba-370M…
R3 detail_obsessive 8/10
api-gemini · gemini-3.8-flash · conf 5

This paper introduces Mamba, an attention-free sequence modeling architecture based on selective structured state space models (SSMs). The authors demonstrate that linear time-invariant (LTI) SSMs fail at content-based reasoning tasks (e.g., selective copying and induction heads) and resolve this by parameterizing SSM matrices (B, C, and step size Δ) directly as functions of the input tokens. To make input-dependent recurrent computation tractable without memory-bandwidth bottlenecks, the authors design a…

Top weaknesses
  • Absence of variance reporting and statistical significance: Not a single table (Table 1, Table 3, Table 4, Table 5, Table 6, Table 7, Table 8, Table 9, Table 10, Table 13) reports standard deviations, confidence…
  • Non-monotonic downstream scaling in DNA classification: In Table 13 (Great Apes DNA Classification), performance for Mamba-1.4M does not scale smoothly with context length: accuracy is 31.47% at sequence length 2^10,…
  • NLL vs. generative metric divergence on SC09: In Table 4, when scaling Mamba from 6.1M to 24.3M parameters, the validation NLL actually degrades slightly from 1.852 to 1.860, despite generation FID improving from 0.94…

Meta-review

AC estimate: 8/10

## Summary The submission proposes **selective SSMs (S6)**: the SSM parameters ∆, B and C become functions of the input (Algorithm 2, Section 3.2). Because this breaks the convolutional form used by prior LTI SSMs, the authors add a **hardware-aware fused scan** that keeps the expanded (B,L,D,N) state in SRAM rather than HBM (Section 3.3.2). They also propose a homogeneous **Mamba block** that merges the H3 block and the gated MLP (Section 3.4, Figure 3). The evaluation covers: - synthetic tasks (Tables 1, 2/11); - Pile language modeling, with scaling laws to ~1.3B (Figure 4, Table 12) and zero-shot evaluation to 2.8B (Table 3); - DNA (Figure 5, Table 13); - audio (Figure 7, Tables 4–5); - efficiency (Figure 8, Table 15); - ablations (Tables 6–10). All three reviewers judge the paper above the bar: R1 rates it 8, R2 rates it 6 and R3 rates it 8. No reviewer identified a fatal technical flaw, and my own reading found none. ## Strengths (consensus, evidence-backed) 1. **The core idea is clearly isolated by controlled ablations.** - Table 1: moving from S4 to S6 raises selective-copying accuracy from 18.3 to 97.0 (no gate) and from 57.0 to 99.7 (H3 architecture). The Mamba architecture with S4 reaches only 56.4, so the architecture alone does not solve the task. - Table 6: every LTI variant sits between 10.24 and 10.75 perplexity, while S6 gives 8.95 (H3) and 8.69 (Mamba). - Table 10: growing the state size N from 1 to 16 helps only when B and C are selective (9.73→8.71, versus 9.88→9.81 when they are not). This directly supports the compression/state-expansion argument of Sections 3.1 and 3.3. 2. **The systems contribution is real and necessary.** Without the fused scan, the time-varying model would be impractical. Section 4.5 and Figure 8 report the scan…

Revision plan

  1. must Headline claims exceed the tested regime. Scaling laws stop at ~1.3B parameters and 26B tokens (Table 12), and zero-shot evaluation stops at 2.8B (Table 3). The 'matches Transformers twice its size' claim relies on…
    → Rewrite the Abstract, Introduction and Conclusion to state the parameter and token scale explicitly. Phrase 'twice its size' as relative to the Pythia-6.9B and RWKV-7.4B models that share Mamba's data and tokenizer.… R1, R2, R3
  2. should Transformer++, the strongest baseline in Figure 4, is missing from the downstream comparison in Table 3. Rows also mix tokenizers, datasets and context lengths (RWKV was trained at 1024 context, Section 4.2.2), which…
    → Train Transformer++ at ~1.4B (and ideally 2.8B) on the same Pile tokens, tokenizer and context length as Mamba, then evaluate it with the same harness in Table 3. If compute does not allow this, state the omission… R1, R2
  3. should No seeds, standard deviations or confidence intervals are reported in any table. Small margins include Table 3 (Mamba-130M vs Pythia-160M: Arc-C 24.3 vs 24.1, WinoGrande 51.9 vs 51.9), Table 8 (8.71 vs 8.85), Table 9…
    → Report 3–5 seeds for the 350M ablations (Tables 6–10), the SC09 results (Tables 4–5) and the DNA classification results (Table 13). Report bootstrap confidence intervals for the zero-shot scores in Table 3. In the… R1, R2, R3
  4. should Table 13 is non-monotonic and undercuts the long-context claim at intermediate lengths. Mamba-1.4M scores 31.47 / 27.50 / 27.66 at 2^10 / 2^12 / 2^14, near 20% chance. HyenaDNA wins at 2^12 (28.43 vs 27.50), 2^14…
    → Rerun Table 13 with multiple seeds and report the mean and standard deviation per cell. Diagnose fine-tuning instability at intermediate lengths, for example learning-rate sensitivity or sequence-length warmup. Scope… R1, R2, R3
  5. should The selection and compression motivation (Section 3.1) is tested only synthetically (Tables 1, 11). The language evaluation consists of zero-shot multiple-choice tasks and perplexity at 2K context (Section 4.2.2).…
    → Add language-scale probes: multi-query associative recall, needle-in-a-haystack or phonebook lookup, and few-shot ICL at 1.4B–2.8B. Report the hybrid Mamba+MHA results from Appendix E.2.2 at 350M–1.3B. Add the… R1, R2, R3
  6. should Efficiency claims lack operating context. The 4–5× throughput comes from larger batches (Section 4.5), and the 'Mamba-6.9B (untrained)' figure is an extrapolation. Table 15 shows higher training memory than…
    → Report end-to-end training tokens/s versus a FlashAttention-2 Transformer at 2K and 8K context. Report batch-1 decoding latency and throughput across context lengths and batch sizes, with the hardware and precision… R1, R2, R3
  7. could Novelty relative to gated linear RNNs (QRNN, SRU, S5; Section 2) is not isolated experimentally. Theorem 1 covers only N=1, A=−1, B=1.
    → Add an ablation comparing a gated linear RNN with N=1 (no state expansion) against S6, both run on the same fused scan and block. This separates the effect of selectivity from the effect of state expansion, building on… R1
  8. could Audio results have unexplained details. Table 4 shows NLL worsening from 1.852 to 1.860 while FID improves from 0.94 to 0.67 when scaling from 6.1M to 24.3M. Audio alone switches to a complex parameterization (Section…
    → Explain the NLL/FID divergence, for example overfitting on SC09 or a sampling-temperature effect. Bring the real-versus-complex audio ablation from Figure 10 into the main text. Optionally run complex versus real on a… R1, R3