Event
Ph.D. Research Proposal Exam: Maryam Maghsoudi
Tuesday, November 24, 2026
10:30 a.m.
AVW 2460
Announcement: Ph.D. Research Proposal Exam
Name: Maryam Maghsoudi
Committee:
Professor Shihab Shamma (Chair)
Professor Behtash Babadi
Professor Jonathan Z. Simon
Date/time: Tuesday, November 24 at 10:30 AM
Location: AVW 2460
Title: From Perception to Imagination: Understanding and Decoding Speech Representations in the Human Brain
Abstract: Imagined speech is the process of internally generating speech without producing any sound or articulatory movement. Decoding imagined speech from neural activity could provide a means of communication for individuals who are unable to speak, such as patients with severe motor disabilities. However, imagined-speech decoding is challenging. Unlike perceived or produced speech, neural responses during imagination are generally weaker. In addition, imagined speech has no observable acoustic output, making it difficult to determine the timing and content of imagined words. The limited availability of imagined-speech datasets also makes training reliable decoding models challenging. Previous studies have shown that speech perception, production, and imagination involve partially shared neural representations. This motivates the use of these relationships to develop imagined-speech decoding frameworks.
In this dissertation, we develop an imagined-speech decoding framework that uses the relationship between neural responses during listening and imagination. The framework consists of two models: a mapping that predicts listened neural responses from imagined responses and a speech decoder trained only on listened responses. To decode imagined speech, we first use the mapping to predict the corresponding listened response, which is then given to the decoder to retrieve information about the imagined speech. This allows us to decode imagined speech without training the speech decoder directly on imagined responses.
First, we collected a paired magnetoencephalography (MEG) dataset to study the relationship between neural responses during listening and imagination. The dataset includes recordings from trained musicians during listening and imagination of rhythmic poems and musical melodies. The use of rhythmic stimuli and recruitment of musicians helped maintain consistent timing during imagination. The dataset includes multiple repetitions of each stimulus under both listening and imagining conditions. This paired structure allows us to relate neural responses to the same stimuli during listening and imagination.
Second, we developed computational models to predict neural responses during listening from corresponding responses during imagination. We show that the relationship between these conditions can be learned and generalized to unseen participants. We also show that the predicted listening responses preserve stimulus-specific information.
We also investigated shared neural representations across speech production conditions using stereotactic EEG (sEEG) recordings during vocalized, mimed, and imagined speech. We used linear spectrotemporal decoding models to reconstruct speech features from neural activity and showed that models trained on one condition can generalize to another.
Finally, we developed a contrastive learning model for decoding speech from listened MEG responses and applied it to imagination using the learned mapping. The decoder is a multimodal contrastive decoder that aligns neural activity with acoustic and linguistic representations of speech. We evaluate the model using word retrieval accuracy and rank-based measures and show that the performance is significantly above our shuffled baseline.
This work therefore proposes a framework for developing imagined-speech decoders that make use of information from other speech conditions and does not require direct training on imagined neural responses.
