Dr. Shuhao Cao has received a prestigious National Science Foundation CAREER Award for his project, “CAREER: Understanding the Inner Mechanism of Transformer Neural Operators.”

Transformers are the neural network architecture underlying many of today’s most widely used artificial intelligence systems, including large language models, computer vision tools and scientific AI applications. Despite their rapid success, much about how Transformers internally represent and process information remains difficult to explain.

Cao’s project will develop new mathematical tools to better understand the “latent space” inside Transformer models, helping make these powerful systems more interpretable. His approach draws on approximation theory from partial differential equations, or PDEs, to study how hidden representations are formed and propagated through Transformer layers.

At the center of the project is the Transformer’s attention mechanism. Cao will analyze attention as a nonlinear operator and study Transformers as function-to-function maps in infinite-dimensional mathematical spaces. The goal is to develop rigorous ways to measure what these models can represent and how their internal structure contributes to that capability.

This work contributes to the broader effort to move advanced neural networks from “black-box” systems toward more interpretable and explainable models, an important step in the development of safer and more trustworthy AI.

The project also includes two applied and educational components. Cao will explore how Transformer-based signal processing may improve the musical experience of people with disabling hearing loss, including users of cochlear implants and receiver-in-the-ear hearing aids. He will also share approaches for redesigning assessment in scientific computing courses in response to the growing use of generative AI, helping inform how future computational mathematicians are trained.

The NSF CAREER program supports early-career faculty who have the potential to serve as academic role models in both research and education. Cao’s award reflects the growing intersection of mathematics, artificial intelligence and scientific computing, while creating new opportunities to understand and improve the technologies increasingly shaping modern science and society.