HyperMLP: An Integrated Perspective for Sequence Modeling
ICML 2026
Jiecheng Lu
Ph.D. candidate in Machine Learning at Georgia Tech (ISyE), advised by Prof. Shihao Yang.
I work on how sequence models read context, store knowledge, and rewrite their own weights, toward self-improving LLMs, agents, and AGI.
Today’s foundation models stop learning when training ends. I study parametric memory, the knowledge a model keeps in its weights, as the route to continual learning, LLM self-improvement, and recursive self-improvement in agentic systems.
It builds on my architecture work, which makes sequence models for LLMs and time series more expressive under practical compute constraints, across attention, linear attention, and dynamic MLPs. Since 2024, I have published 10 papers as the main contributor at ICLR, ICML, and NeurIPS.
News
All newsResearch
All publicationsOne question runs through my work: how a model holds what it has seen and puts it to use. The papers below answer it in the context; my current work asks it of the weights, as noted under each line, along with how pretraining stores knowledge there and why that stored knowledge can override the context.
NextGen sequence modeling
LLM pretraining, efficient and expressive attention, and dynamic MLPs
Dynamic MLPs already instantiate weights from the context; I now ask how a model can keep such writes in its own weights once the context is gone.
Free Energy Mixer
ICLR 2026
ZeroS: Zero‑Sum Linear Attention for Efficient Transformers
NeurIPS 2025
Spotlight
Attention meets dynamical systems
VAR/ARMA attention and symplectic position encoding beyond RoPE
ARMA attention already models the errors of its own past predictions; I now use error-driven updates of this kind to write new knowledge into a model's weights.
Forecasting for real-world decisions
In-context, multivariate, and distribution-valued time series
ICTSP already learns to forecast from examples in its context; I now ask when what a model learns in context should move into its weights.
CAST: Causal Anchored Simplex Transport for Distribution-Valued Time Series
NeurIPS 2026
In‑context Time Series Predictor
ICLR 2025
CATS: Enhancing Multivariate Time Series Forecasting by Constructing Auxiliary Time Series as Exogenous Variables
ICML 2024
ARM: Refining Multivariate Forecasting with Adaptive Temporal‑Contextual Learning
ICLR 2024
Research translation
CDC FluSight forecasting hub
Fall 2024–PresentSubmit weekly influenza hospitalization forecasts for all U.S. states to the CDC FluSight forecasting hub (entry name: Gatech-ensemble), translating time-series forecasting research into public-health decision support.
Co-author of Multi-season evaluation and analysis of categorical trend forecasts of influenza hospital admissions in the United States (medRxiv preprint, 2026; submitted to Nature Communications).
* Corresponding author.
Talks
All talks Rethinking Sequence Modeling with HyperMLP: An Integrated Architectural Perspective
Mar 2026Invited Talk, Tsinghua University, Knowledge Engineering Group
Hosted by Prof. Jie Tang, founder of Z.ai
Research leadership and mentoring
Since Fall 2024, I have organized a Georgia Tech undergraduate Sequential AI team, leading each project through the full research cycle, from idea to publication.
Yubin Kim ZeroS (NeurIPS 2025), StretchTime (ICML 2026); 1 ICLR 2027 submission now Ph.D. student in OR, MIT
Viresh Pati ZeroS (NeurIPS 2025), StretchTime (ICML 2026); 1 ICLR 2027 submission now Ph.D. student in CS, UPenn
Teaching
ISyE 4031 Regression and Forecasting
Summer 2026Independent Instructor, Georgia Institute of Technology
Three sections (one in person, two online), 58 students in 28 project teams. Participant in Georgia Tech's Tech to Teaching program; completed CETL 8717 Course Design for Higher Education.
Honors and grants
All awardsFellowships and awards
George Fellowship, Georgia Tech 2026
Margaret and Stephen Kendrick PhD Student Fellowship for Research Excellence, Georgia Tech 2025
M.H. Stewart Fellowship, Georgia Tech 2023
Computing grants
Lambda Research Grant, PI
Mar 2026–Mar 2027Subquadratic HyperMLP via Convolution-Based Sequence Mixing
NVIDIA Academic Grant, awarded project
Apr 2026–Sep 2026Scaling Zero-Sum Linear Attention for Cross-Modal Foundation Models
NSF ACCESS Program, awarded project
Sep 2025–Sep 2026Advanced Transformer-based Models for Long-term Time Series Forecasting