Free Energy Mixer
Introduces Free Energy Mixer (FEM), which interprets (q,k) attention scores as a prior and performs a log-sum-exp free-energy readout to reweight values at the channel level, enabling a smooth transition from mean aggregation to selective channel-wise retrieval without increasing asymptotic complexity.
Free Energy Mixer (FEM) replaces head-level convex combination with a free-energy-based readout that combines prior attention and value evidence, endowing attention with channel-wise selectivity. FEM is plug-and-play for standard and linear attention, RNNs, and SSMs, improving expressiveness and performance while preserving computational efficiency.