Transformers are inherently succinct, paper argues

A paper posted on OpenReview on June 5, 2026 argues that the transformer architecture’s self‑attention yields a provably compact representation of certain function classes [OpenReview]. The authors prove that a transformer with O(log n) layers can encode the same information as a recurrent network that requires O(n) steps, establishing an inherent succinctness property for sequence lengths n.

The proof hinges on three architectural features. First, self‑attention aggregates information across the entire sequence in a single layer, eliminating the need for deep recurrence. Second, weight sharing across attention heads reduces the parameter count needed to capture long‑range dependencies. Third, learned positional encodings preserve order information without inflating the model size. Together these mechanisms enable transformers to represent complex mappings with fewer layers and parameters than comparable architectures.

Why it matters:

• Training and inference cost – Fewer layers translate directly into lower FLOP counts, cutting both GPU time and energy consumption.
• Model size – The compact representation allows practitioners to prune or quantize models without sacrificing accuracy, facilitating deployment on edge devices and mobile platforms.
• Research direction – Demonstrating a theoretical bound for transformer succinctness may shift focus toward architectures that exploit this property, rather than pursuing ever‑larger models.

The paper’s results provide a concrete foundation for ongoing work in model compression and efficiency, suggesting that architectural choices, not just scale, can drive performance gains.