Two UMD PH.D. Students Awarded Qualcomm Innovation Fellowship

Two students from Professor Ang Li’s Collaborative, Automated, Scalable, and Efficient Intelligence (CASE) Lab have been awarded theQualcomm Innovation Fellowship (QIF). ECE student Guoheng Sun and Computer Science student Shwai He are one of only 17 teams receiving the 2025 award. This marks the first time a team from UMD has received this distinguished award.

The QIF is a research funding and mentorship program sponsored by Qualcomm, a leading company in semiconductors, software and wireless technology. The program recognizes and supports graduate students who have been recognized for their exemplary technical research. Each winning team shares a $100,000 prize to support their research and academic pursuits.

"I’m tremendously proud of Shwai and Guoheng for receiving the QIF Innovation Fellowship,” says Professor Ang Li, ECE Assistant Professor and Affiliate Assistant Professor in Computer Science. “Their work challenges foundational assumptions in Transformer architecture and opens new possibilities for building faster, more efficient language models. This recognition reflects not only their technical creativity but also their commitment to impactful and forward-thinking research."

Shwai and Guoheng's award-winning proposal, "Less Attention, Much Faster: Toward a Future of Efficiency-Optimized Transformer Architectures," addresses the computational inefficiency of Transformer models, which are central to Large Language Models (LLMs). Their prior work introduced "Attention Drop," a method to significantly speed up inference by removing redundant self-attention modules without major performance loss.

Their QIF-funded research will build upon this foundation, focusing on:

• Dynamic Attention Drop: Investigating methods to intelligently and dynamically skip attention layers based on real-time input demands.

• Training-Aware Attention Drop: Integrating attention-dropping strategies directly into the model training process to create inherently leaner and more efficient architectures.

• Optimization of Mobile Inference: Tailoring these efficient models for deployment on resource-constrained mobile devices, aiming to enhance speed and reduce energy consumption, potentially leveraging Qualcomm's hardware capabilities.

Regarding the award, He notes “The QIF Fellowship will enable us to further investigate the surprising redundancy in attention mechanisms, a central component of modern LLMs. With Qualcomm's support, we hope to develop practical solutions that dramatically accelerate inference without sacrificing accuracy—making large models more accessible and deployable.”

Sun adds, “This fellowship gives us the momentum to push our research into dynamic and training-aware attention pruning. We aim to explore novel architectures that scale more efficiently, and Qualcomm's guidance will be instrumental in bridging theory with real-world deployment constraints."

The students also expressed deep gratitude for the invaluable guidance provided by their advisor, Professor Ang Li,, as well as the generous support from the CASE lab team. Professor Li, whose research focuses on efficient AI, played a crucial role in shaping their research direction and fostering their innovative approach.

The prestigious Qualcomm fellowship not only recognizes the groundbreaking potential of Shwai and Guoheng’s research but also provides crucial resources and mentorship to propel their work forward, contributing to the development of more efficient and accessible next-generation AI.

Published June 9, 2025