Five sequence architectures, implemented from first principles and measured
Pure-PyTorch implementations of S4D, Mamba, Mamba-2, Mamba-3 (complex) and a Transformer baseline. Parallel and recurrent forms are proven equivalent by tests; every number below is a committed measurement, not a paper claim.
How each model reads a stream
A simplified walkthrough of what each architecture does as tokens arrive — teaching schematic, not a simulation of the real kernels. Pick a model, then play or step through. The interesting differences: what the model keeps (state), what it filters (selectivity), and how much compute each step costs.
Capability: where each architecture wins
Cumulative-parity tracking needs rotation; only complex, input-dependent transitions can represent it. Selective copying needs gating; LTI models cannot filter noise. Exact copying needs memory that grows with length, where attention dominates. The ranking flips with the task - that is the point.
Parity / state tracking (token accuracy)
Cumulative XOR, L=128, 20 epochs, matched budgets. Dashed line = chance (0.5).
Selective copying
Marker-value pairs in noise; train L=64 (3–5 pairs), test L=128 (5–10 pairs). Two bars per model: full-sequence success and token accuracy. Mamba-3 trades recall sharpness for state tracking at this budget - reported as measured.
Exact copying (in-distribution L=32)
Reproduce a 32-token string (Jelassi et al., ICML 2024 formulation). The Transformer-vs-SSM ranking from the paper reproduces; the same models flip ranking on selective copying above.
IMDb sentiment (accuracy / AUC)
12.5k train samples (fixed seed), max length 512, 3 epochs. Mamba-1/3 are intentionally not trained here: their python-loop scans are wall-clock-prohibitive (see efficiency charts); Mamba-2 shares their recurrence.
Same models, different task: the ranking flips
One of the study's central results. Switch the task and watch the ordering change — no architecture dominates; the bottleneck (rotation / gating / linear memory) decides. measured result
Efficiency vs sequence length (1K–32K)
Measured on this machine's 6 GB RTX 4050 (fp32, batch 8 for train-step charts, CUDA events, 5 repeats). Drag the slider to jump between measured lengths — every readout snaps to a committed data point; nothing in between is interpolated. × markers are measured out-of-memory limits, not missing data.
Training step time (ms, log-log)
Peak training memory (MB)
Training throughput (tokens/s)
Decode cost per token (ms)
Decode modes are labeled per model: Mamba-1/S4D use true O(1)-state stepwise decode; Mamba-2/3 report sequential-scan cost; the Transformer has no KV cache and recomputes the full prefix per token.
How was this measured?
The benchmark is an experiment, not a stopwatch screenshot. Step through the actual protocol implemented in benchmarks/efficiency.py: measured methodology
- Correctness is tested, not asserted: S4D's FFT-convolution form equals its recurrent step form; Mamba-2's chunked SSD equals its sequential reference scan; Mamba-3 reduces to the Mamba-1/2 update when λ=1 and rotation frequency is zero; decode paths equal parallel paths. Run:
python -m pytest tests/ -q - Every figure on this page regenerates from committed JSON:
results/*.json,python demo-site/build.py. - Theoretical O(T) and wall-clock are different claims; the python-loop scans are O(T) in operations but 10–100× slower in wall-clock - exactly why fused kernels and Mamba-2's matmul form exist.
- Full methodology, result tables and literature reproductions: technical report (PDF) and the README.