Dr. Patrick Rohrer (Universitat Rovira i Virgili)

Face-to-face communication is a multimodal affair, where interlocutors leverage multiple signals from the voice (propositional content, speech prosody) and the body (manual gestures, facial expressions, and head movements) to convey and comprehend meaning. From the perspective of multimodal prosody and the modality-neutral hypothesis—which posits that the basic notions of prominence (relative salience) and phrasing (chunking) surface both in the acoustic domain as speech prosody and in the visual domain through gestural prominence and beat-like-ness—this talk explores how these two modalities interact in the perception of prominence.

The presentation will begin with a brief introduction to the multimodality of language and the relationship between co-speech gestures and speech prosody, followed by a review of the main findings in the literature on multimodal speech perception. It will then turn to recent studies that approach this question from a cross-linguistic perspective, examining how prosodic typology may shape multimodal speech perception. Taken together, these results suggest that while the two modalities closely interact in the perception of prominence, this interaction unfolds in a language-specific manner.

Dr. Celia Gorba (Universitat Autònoma de Barcelona)

This talk will present results of a Multimodal High Variability Perceptual Training for learning L2 English intonation. The effects of Phonological vs Pragmatic-based Gestural Support will be assessed.  Three HVPT training regimes will be compared: one with a hand gesture mimicking pitch direction, one with a facial gesture conveying the pragmatic meaning of the stimulus, and a control group with no gestures.