Perception difficulties and errors in multimodal speech: the case of vowels

TL;DRAbstract

This study is motivated by the increased need to understand human response to video-links, 3G telephony and avatars as screen embodiments of software agents. We focus on response of participants to audiovisual presentations of talking heads, and examine the effect of noise and temporal misalignment of channels in vowels. We confirm that incongruence of audio and visual channels causes strange perception phenomena (the socalled McGurk fusion) but here we concentrate on such effects in vowels. We show that incongruent vowel stimuli elicit fusion even in the presence of noise, and that fusion is more common in short vowels than long. In all cases of stimuli with incongruent channels, subjects are found to take more time to reach decisions than in the case of congruent stimuli. Where fusion is reported, decision time is significantly lower than in cases where the subject decides for the visual or for the acoustic stimulus. It is argued that decision times – which we measure in two differen

Chat with Paper

AI Agents for this Paper