HyperMLP: An Integrated Perspective for Sequence Modeling
Presents an integrated dynamic-MLP perspective on sequence modeling, reinterpreting attention heads through context-instantiated MLP computation and learnable sequence-space mixing.
HyperMLP develops an integrated architectural view of sequence modeling by interpreting autoregressive attention heads as dynamic MLPs whose weights are instantiated from context history. The method introduces learnable sequence-space mixing aligned with autoregressive semantics to improve expressive routing under practical compute constraints.