Joint EEG/fMRI on Extracting Integrated Semantic and Syntactic Information
Description
The imaging modalities used in the many recent studies suffer from complementary limitations of both technologies; EEG has a poor spatial resolution and fMRI has a low temporal resolution. An important aspect of this research project is the use of simultaneous EEG and fMRI registration on the semantic and syntactic information processing in the brain which will allow to improve the characterization of cognitive processes by combining the strengths of both techniques. The study aims to extend and apply the very promising recent approaches to fMRI analysis based on multi-voxel patterns to joint EEG/fMRI recordings. Methods of characterizing functional connectivity patterns from such recordings will then be used to establish the nature of integrated linguistic representations.
Initial experiments will be conducted to investigate the brain representations of analytic semantic categories. Further experiments will present particular syntactic constructions in which the words will be drawn from the semantic categories established from earlier experiments. These will be paired with sequences of the same words presented in a random (non-grammatical) order. The regularities of brain activations will be investigated by comparing the brain signals across these conditions. A model based on geometric representations combining syntactic and semantic information will be trained on EEG/fMRI data which with the aim of developing a decoder that can predict properties of the sentences based on aspects of the brain registration.
In addition to revealing fundamental properties of information encoding in brain signals, this research will have significant implications for language parsing systems which could be used as part of the assistive technology augmenting communication for subject with linguistic impairments.
Project Research Group
Slawomir Nasuto
Professor of Cybernetics, University of Reading
James Douglas Saddy
Professor of Clinical Language Sciences, University of Reading
Isil Bilgin
PhD Student, School of Systems Engineering, University of Reading