Reordered MoE checkpoint cuts SSD reads 2.23× and speeds decode 32% on 235B model

Reordering the expert weights inside a Mixture‑of‑Experts (MoE) checkpoint reduced SSD reads 2.23× on an 80B model and added a 32.3% decode‑throughput gain on a 235B model running on a 48 GB MacBook, measured by an unrelated engine author using the original script unchanged [DevTo].

What shipped

The mbolt project applied BOLT/PGO‑style binary rewriting to MoE checkpoints. By tracing routing, clustering co‑activation, and permuting the weight file byte‑exactly, the layout changed from thousands of scattered reads per token to a handful of sequential reads. On a solo run the author recorded a 2.23× read‑reduction and a 1.55× end‑to‑end decode gain against a patched llama.cpp baseline.

Independent verification

• @mabaeyens measured the same 2.23× read reduction on the mlx‑lm expert‑offload streamer, confirming the layout effect on an explicit‑read engine [MLX-LM].
• @lBroth applied the unmodified conversion script to a 119 GB Qwen3‑235B build and observed a 32.3% decode boost on his own engine, matching the original claim [DevTo].
• @philipjohnbasile ran the colibri 744B engine, which already stores each expert contiguously, and saw only a +1.3% residual gain, showing that the optimization collapses when the baseline read pattern is already optimal [Colibri].

Why it matters

The reordering only benefits engines that issue explicit reads; mlx‑lm and colibri saw double‑digit gains, while stock llama.cpp, which relies on mmap, showed no improvement. Community‑driven verification turned a speculative idea into a reproducible law for slice‑size gains, letting engineers predict ROI based on cold‑read share and I/O latency. This quantitative model helps decide whether the engineering effort of reordering outweighs expected performance gains for a given hardware class.

Reader poll

Which inference‑engine design philosophy do you back?

• Explicit‑read expert streaming (mlx‑lm, colibri)
• Opaque black‑box inference (stock llama.cpp)
• Hybrid prefetch + LRU eviction
• No‑change, focus on model scaling