Skip to main content

Kimi Team's 'Attention Residuals' Revolutionize Transformer Efficiency and Performance

23 MARCH 2026·6 MIN READ·5 SOURCES

The Kimi Team introduces Attention Residuals, a novel architectural replacement for fixed residual connections in Transformers, leveraging softmax attention to achieve a 1.25x compute efficiency gain and significant performance boosts on reasoning tasks with minimal overhead.

Kimi Team's 'Attention Residuals' Revolutionize Transformer Efficiency and Performance

Key takeaways · 5

  • 01

    Attention Residuals replace fixed additive residual connections with a learned softmax attention mechanism over previous layers.

  • 02

    This approach mitigates the PreNorm dilution problem causing uncontrolled growth in hidden state magnitudes with depth.

  • 03

    Block AttnRes partitions layers into blocks, attending only to block-level summaries, reducing memory costs with minimal performance loss.

  • 04

    Kimi Team reports a 1.25x compute efficiency advantage, equivalent to training baselines with 25% more compute.

  • 05

    Performance gains include +7.5% on GPQA-Diamond and +3.1% on HumanEval benchmarks, important for reasoning and coding capabilities.

Background: Residual Connections and Their Limitations

Traditional residual connections in Transformers add outputs from previous layers uniformly, which leads to hidden-state magnitude growing linearly with depth, known as PreNorm dilution. This dilutes early layer contributions and limits effective depth.

Attention Residuals: Concept and Mechanism

Attention Residuals (AttnRes) replace the fixed summation with content-dependent softmax attention, allowing each layer to selectively attend to earlier representations. This is achieved using learned pseudo-queries per layer and keys/values from preceding layer outputs.

Scaling and Efficiency with Block AttnRes

To address quadratic memory overhead of full AttnRes, Block AttnRes partitions Transformer layers into blocks and only attends over block-level summaries. This design recovers most performance gains while adding minimal computational and memory overhead.

Empirical Results and Benchmarks

On compute-equivalent comparisons, AttnRes matches baselines trained with 25% more compute, delivering +7.5% performance on GPQA-Diamond and +3.1% on HumanEval. It shows consistent improvements across model sizes and various task domains.

Residual connections are a fundamental and almost untouched component in Transformer architectures. The Kimi Team's Attention Residuals provide an elegant solution to overcome inherent limitations of fixed residual summation, delivering significant compute efficiency improvements and better performance on complex reasoning tasks with negligible inference overhead. This breakthrough paves the way for more resource-efficient, scalable, and capable large language models, offering immediate practical benefits for AI research and industry deployments.

Why it matters
Daily session

Put this to work — one session a day, built for your industry.

Create a free account for a daily session — eight questions and one real-work challenge, on the news that affects your role.

Start free

Sources

AI fluency, one session a day, built for your work.

Get the appStart free