Event
ECE Special Seminar - Aditya Ramamoorthy, Iowa State University
Wednesday, February 19, 2025
4:00 p.m.
2460 A.V. Williams
Darcy Long
301 405 3114
dlong123@umd.edu
Speaker: Professor Aditya Ramamoorthy, Iowa State University
Talk Title: Efficient gradient coding for mitigating stragglers within distributed machine learning
Abstract: Large scale distributed learning is the workhorse of modern-day machine learning algorithms. A typical scenario consists of minimizing a loss function (depending on the dataset) with respect to high-dimensional parameter. Workers typically compute gradients on their assigned dataset chunks and send them to the parameter server (PS), which aggregates them to compute either an exact or approximate version of the overall gradient of the relevant loss function. However, in large-scale clusters, many workers are prone to straggling (are slower than their promised speed or even failure-prone). A gradient coding solution introduces redundancy within the assignment of chunks to the workers and uses coding theoretic ideas to allow the PS to recover the overall gradient (exactly or approximately), even in the presence of stragglers. Unfortunately, most existing gradient coding protocols are inefficient from a computation perspective as they coarsely classify workers as operational or failed; the potentially valuable work performed by slow workers (partial stragglers) is ignored.
In this talk we will give an overview of some of our recent work in this area that addresses these limitations. Specifically, we will present novel gradient coding protocols that judiciously leverage the work performed by partial stragglers. Our protocols are simultaneously efficient from both a computation and communication perspective and numerically stable. For an important class of chunk assignments, we present efficient algorithms for optimizing the relative ordering of chunks within the workers; this ordering affects the overall execution time. For exact gradient reconstruction, our protocol is around 2x faster than the original class of protocols and for approximate gradient reconstruction, the mean-squared-error of our reconstructed gradient is several orders of magnitude better.
Bio: Aditya Ramamoorthy is a Professor of Electrical and Computer Engineering and (by courtesy) of Mathematics at Iowa State University. He received his B. Tech. degree in Electrical Engineering from the Indian Institute of Technology, Delhi and the M.S. and Ph.D. degrees from the University of California, Los Angeles (UCLA). His research interests are in the areas of classical/quantum information theory and coding techniques with applications to distributed computation, content distribution networks and machine learning.
Dr. Ramamoorthy currently serves as an editor for the IEEE Transactions on Information Theory (previous term from 2016 — 2019) and the IEEE Transactions on Communications from 2011 — 2015. He is the recipient of the Northrop Grumman professorship (2022 - 204), the 2020 Mid-Career Achievement in Research Award, the 2019 Boast-Nilsson Educational Impact Award and the 2012 Early Career Engineering Faculty Research Award from Iowa State University, the 2012 NSF CAREER award, and the Harpole-Pentair professorship in 2009-2010.
