Prebuttal

Mamba: Linear-Time Sequence Modeling with Selective State Spaces

ICLR · simulated review · 2026-10-27T07:37:30+00:00 · seed 42
backends: R1: agent-claude · R2: agent-codex · R3: api-gemini
Unofficial demo: an AI-simulated review of arXiv:2312.00752v2 by Albert Gu and Tri Dao, licensed CC BY 4.0. This is arXiv v2 (May 2024) — the revised version, accepted at COLM 2024. Summaries and short quotes below were generated by AI models from the paper and may contain errors; generated only to show what Prebuttal outputs. Not written, reviewed or endorsed by the authors, and not an assessment of the work's merit. Authors who want this page removed: open an issue.
7.33/10
estimated · σ 0.94 · range 6–8 · n=3
Three independent AI reviewers, isolated contexts, plus an area chair. This is a pre-submission stress test, not a prediction.
R1 senior_kind 8/10
agent-claude · sonnet · conf 4

The paper argues that a key weakness of subquadratic sequence models (LTI structured SSMs, global convolutions, linear attention) is their inability to perform content-based selection, i.e., to propagate or forget information depending on the current token. It makes the SSM parameters (Δ, B, C) input-dependent (S6), which breaks the convolutional computation path, and compensates with a hardware-aware parallel scan that avoids materializing the (B, L, D, N) state in slow GPU memory. This is wrapped in a…

Top weaknesses
  • Claims of 'state-of-the-art across modalities' and 'first linear-time model that truly achieves Transformer-quality' (Abstract, Section 1, Conclusion) are stronger than the evidence, which stops at ~1.3B for scaling…
  • Core capabilities where attention is known to excel are untested: retrieval/recall-heavy tasks, long-context language tasks, in-context learning at scale, and copying with fixed-size state. The compression framing…
  • Several comparisons are not equally controlled. Baselines RWKV and RetNet are missing at 8k context due to efficiency (Section 4.2.1), Transformers are omitted at long DNA lengths and from audio outer blocks (Table 5…
R2 adversarial 6/10
agent-codex · gpt-6-luna · conf 4

The paper introduces selective state space models, making parameters such as Δ, B, and C depend on the input, and pairs them with a hardware-aware scan to avoid materializing the full expanded state. It combines these layers with gated projections in a homogeneous architecture, Mamba, and evaluates the result on synthetic tasks, language, DNA, audio, and efficiency benchmarks. The reported results include strong language-modeling scores at up to 2.8B parameters, improved long-context DNA and audio results, and…

Top weaknesses
  • The broad foundation-model motivation is stronger than the demonstrated scale and breadth. Section 5 explicitly says language evaluations stop below the 7B scale of strong open models, while the largest trained Mamba…
  • The closest architectural comparisons are not consistently controlled. Section 4.2 compares against Transformer++ but notes that RWKV and RetNet lack 8k-context results because of implementation constraints; Table 3…
  • The headline efficiency claim needs tighter scope. Section 4.5 reports 4–5× inference throughput, attributing it to larger feasible batch sizes without a KV cache, while Table 15 shows Mamba using more memory than…
R3 detail_obsessive 8/10
api-gemini · gemini-3.8-flash · conf 5

This paper introduces Mamba, a sequence modeling architecture based on selective structured state space models (SSMs). The core contribution is making the SSM parameters (Δ, B, C) input-dependent, which equips linear-time recurrent models with content-based filtering and context-compression capabilities necessary for discrete data like text and DNA. To overcome the resulting loss of time-invariance (which previously enabled global convolution), the authors design an exact, hardware-aware parallel associative scan…

Top weaknesses
  • Lack of statistical reporting and error bars: Throughout the paper, including downstream evaluations (Table 3), speech generation metrics (Table 4, Table 5), ablations (Tables 6–10), and DNA classification (Table 13),…
  • Non-monotonic downstream performance in DNA modeling: In Table 13, Mamba-1.4M exhibits erratic behavior at shorter sequence lengths, starting at 31.47% accuracy at sequence length 2^10, dipping to 27.50% at 2^12 and…
  • Training memory footprint remains higher than optimized attention: Although Mamba scales linearly in memory, Table 15 reveals that across all tested batch sizes (1 to 32), Mamba's training memory footprint on a 125M…

Meta-review

AC estimate: 8/10

## Summary of the submission The paper argues that linear-time sequence models (LTI SSMs, global convolutions, linear attention) underperform attention on discrete data because they cannot do content-based selection. It proposes making the SSM parameters Δ, B, C functions of the input (S6, Section 3.2). Because this breaks the convolutional computation path, it adds a hardware-aware scan that keeps the (B, L, D, N) expanded state in SRAM (Section 3.3). The layer sits inside a homogeneous block that merges the H3 block with a gated MLP (Mamba, Section 3.4). Experiments cover: - synthetic tasks (Tables 1, 11) - Pile language modeling, with scaling laws to ~1.3B (Fig. 4) and zero-shot evaluation to 2.8B (Table 3) - HG38 DNA (Fig. 5, Table 13) - audio (Fig. 7, Tables 4-5) - efficiency (Fig. 8, Table 15) - ablations (Tables 6-10) ## Assessment of the evidence All three reviewers agree on the core contribution, and I concur. The synthetic results isolate the mechanism cleanly: - **Selective Copying (Table 1).** S4 reaches 18.3% and S6 with no gate reaches 97.0%. Gating alone does not close the gap: H3+S4 gets 57.0 and Mamba-block+S4 gets 56.4. This directly supports the claim that architectural gating is not selection. - **Induction Heads (Table 11).** A 74K-parameter Mamba trained at length 256 is perfect up to 2^20. MHA variants fall to 5-10% by 2^12-2^14 and run out of memory from 2^15. - **Ablations.** They are well targeted: - Table 6: S6 gives 8.95/8.69 versus 10.24-10.75 for the LTI layers. - Table 7: full selectivity gives 8.71 versus 10.93 with none. - Table 10 is the most convincing mechanistic evidence. Growing N from 1 to 16 barely helps with constant B, C (9.88 to 9.81) but helps a lot with selective B, C (9.73 to 8.71). - **Language modeling (Table…

Revision plan

  1. must Headline claims exceed the evidence. - 'State-of-the-art across several modalities' and 'first linear-time model that truly achieves Transformer-quality' (Abstract, Section 1, Conclusion) rest on models of at most…
    → Rewrite the Abstract, Introduction and Conclusion to qualify the claims by scale (at most 2.8B parameters, 300B tokens, 2K-context downstream evaluation). State the 'twice its size' claim as average zero-shot accuracy.… R1, R2, R3
  2. must Efficiency claims are under-specified. - The 5× inference throughput (Abstract, Section 4.5, Fig. 8 right) relies on larger batches without a KV cache, and one illustration uses an untrained Mamba-6.9B. - Speedups are…
    → In the main text, state the throughput protocol: hardware, precision, prompt and generation lengths, and batch sizes. Report throughput at equal batch size and at matched memory. Say what each speedup is measured… R1, R2, R3
  3. should No seeds, variance or confidence intervals are reported for Table 3 or Tables 4-10, or Table 13. Several claimed effects are small: Table 5 NLL spans 1.850-1.867, Table 8 compares 8.71 vs 8.85, and Table 6 compares…
    → Report multi-seed results with standard deviations for the ablations at 350M (Tables 6-10) and for Table 13 and the SC09 results. For Table 3, give per-task standard errors and acknowledge that pretraining is single-run. R1, R2, R3
  4. should Table 13 is noisy and the 7M model is unmatched. - Mamba-1.4M falls below parameter-matched HyenaDNA at 2^12 (27.50 vs 28.43), 2^14 (27.66 vs 41.17) and 2^16 (40.72 vs 42.22). It wins clearly only at 2^20 (71.67 vs…
    → Add seeds for Table 13. Confirm that fine-tuning hyperparameters were tuned equally per model and per length. Either add a 7M HyenaDNA baseline or present the 7M Mamba row as supplementary. Correct or explain the token… R1, R2, R3
  5. should Fixed-state recall limits and long-context language quality are untested. The compression framing in Section 3.1 implies a recall bottleneck, yet Table 3 downstream evaluations all use 2048 context, and the…
    → Add multi-query associative recall with a growing number of key-value pairs, and evaluate the Fig. 4 8k-context models on long-context perplexity or retrieval tasks. Report hybrid Mamba+attention results (Appendix… R1, R2
  6. should Comparisons with the closest recurrent and gated baselines are incomplete. RWKV and RetNet lack 8k results (Section 4.2.1), RWKV in Table 3 was trained at 1024 context versus 2048, and S5, QRNN and SRU, named in…
    → Add matched-recipe runs at 350M for at least one gated RNN (QRNN or SRU) and a time-varying S5 variant. Give RetNet and RWKV numbers at 8k where feasible, or state the compute limits explicitly. Make clear that the… R1, R2
  7. could The general-backbone claim is in tension with the audio results. Audio is the only experiment that needed complex parameters (Section 4.4.1), and Fig. 10 and Section 5 indicate that selection can hurt on continuous…
    → Move the key Fig. 10 ablation into the main text. Give practical guidance on choosing LTI versus selective layers, or real versus complex parameters, for each modality. R1, R3
  8. could The method depends on a custom fused CUDA kernel tied to the GPU memory hierarchy (Section 3.3). Portability across GPU generations and non-NVIDIA hardware is not discussed.
    → Add a short limitations paragraph on hardware dependence. If available, add benchmark numbers on a second GPU architecture. R1, R3