ZeroS: Zero‑Sum Linear Attention for Efficient Transformers
Introduces Zero‑Sum Linear Attention (ZeroS), which removes the uniform zero‑order term and reweights residuals to enable stable positive/negative attention weights, allowing contrastive operations within a single layer while retaining O(N) complexity.
A zero‑sum reparameterization of linear attention that avoids uniform accumulation bias and allows contrastive operations within a single layer, narrowing the gap with softmax attention under linear complexity.