Δ-Mem cuts memory use in large language models without performance loss

Δ-Mem slashes memory use in large language models by compressing key-value cache states and reusing memory slots during inference, according to an arXiv paper [arXiv]. The method targets the memory bottleneck in autoregressive generation, where models store full attention keys and values for every token—costing gigabytes in long sequences.

The technique works by computing memory deltas—differences between current and prior key-value states—and storing only the changes. It also identifies redundant memory slots using attention similarity thresholds, freeing them for reuse. On Llama-3-8B, Δ-Mem reduced memory consumption by 47% on 32k-token sequences while preserving 100% of the original model’s output accuracy [arXiv].

Unlike static pruning or quantization, Δ-Mem operates online, adapting to input sequences in real time. The authors tested it on code generation and long-document QA tasks, reporting consistent memory savings across both. No performance degradation was observed, even at high compression ratios.

The approach could lower hardware barriers for running state-of-the-art LLMs on consumer GPUs. For example, a model needing 48GB under standard inference ran within 26GB using Δ-Mem—enabling deployment on a single consumer-grade GPU instead of multi-GPU setups.

Existing frameworks like Hugging Face Transformers or vLLM could integrate Δ-Mem with minimal changes, the paper suggests, since it operates at the attention layer level. No retraining is required.