{"id":253,"date":"2019-01-28T21:03:10","date_gmt":"2019-01-29T05:03:10","guid":{"rendered":"http:\/\/linguamonium.com\/?p=253"},"modified":"2021-04-25T05:34:01","modified_gmt":"2021-04-25T05:34:01","slug":"read-my-lips-mcgurk-and-speech-perception","status":"publish","type":"post","link":"https:\/\/linguamonium.com\/?p=253","title":{"rendered":"Read my lips: McGurk and speech perception"},"content":{"rendered":"<p>Think of a situation in which you might try to lip read. Would it be chatting with friends in a noisy bar? Or watching a muted TV? Maybe you were born (or became) deaf. In all of these scenarios, the channel for auditory input is severely obstructed or entirely absent. But what if I told you that lip reading isn\u2019t limited to those contexts? The surprising truth is that we lip read, to some extent, during practically every spoken interaction where we can observe the speaker\u2019s face.<\/p>\n<p>This means that the <em>visual<\/em> information involved in speech perception and processing is way more crucial than we would imagine. For a demonstration, watch <a href=\"https:\/\/www.youtube.com\/watch?v=yJ81LLxfHY8\" target=\"_blank\" rel=\"noopener\">this video.<\/a><a href=\"#_ftn1\" name=\"_ftnref1\">[1]<\/a><\/p>\n<p>As the video mentions, the phenomenon is called the <strong>McGurk effect<\/strong>. It\u2019s named after one of the researchers who first discovered it \u2013 psychologists Harry McGurk and John MacDonald described the effect in their 1976 paper <a href=\"https:\/\/www.nature.com\/articles\/264746a0\" target=\"_blank\" rel=\"noopener\">\u201cHearing Lips and Seeing Voices\u201d<\/a>. They found that when a person watches someone pronounce one sound, but hears the paired audio of another sound, they will perceive a third sound which lies somewhere in-between the visual and auditory cues.<a href=\"#_ftn1\" name=\"_ftnref1\"><\/a><\/p>\n<p>More specifically, the paper details an experiment in which adults watched a video of a woman making the lip\/mouth movements for <em>ga<\/em>, but the audio was dubbed with the syllable <em>ba<\/em>. Subjects reported hearing a third syllable, <em>da<\/em>. (When they watched the undubbed video, they correctly heard <em>ga<\/em>, and when they listed to the audio only, they correctly heard <em>ba<\/em>.)<\/p>\n<p>How is <em>da <\/em>\u201cin-between\u201d <em>ba <\/em>and <em>ga<\/em>?<\/p>\n<p>In phonetics, speech sounds, or <strong>phones <\/strong>(particularly consonants) are classified according to three main parameters: <strong>voicing<\/strong>, <strong>place of articulation<\/strong>, and <strong>manner of articulation<\/strong>. Voicing is whether your vocal folds vibrate when producing the sound \u2013 if they vibrate, the sound is <strong>voiced<\/strong>; if they don\u2019t, it\u2019s <strong>voiceless<\/strong>. Place of articulation deals with how and where your mouth (mostly tongue and lips) moves to constrict or obstruct the vocal tract, creating different consonants. Manner of articulation concerns the way in which air flow passes through or is blocked by the articulators (i.e. mouth\/tongue\/lips) when they\u2019ve taken a given shape.<\/p>\n<p>Identifying such parameters for the <em>ba<\/em>&#8211;<em>da<\/em>&#8211;<em>ga <\/em>example in the 1967 paper:<\/p>\n<ul>\n<li>[Voicing] \/b\/, \/d\/, and \/g\/ are all VOICED consonants\n<ul>\n<li>(Try putting fingers to your Adam\u2019s apple while saying <em>ba<\/em>, <em>da<\/em>, and <em>ga<\/em>, and you should feel the vibration of your vocal folds)<\/li>\n<\/ul>\n<\/li>\n<li>[Manner of articulation] \/b\/, \/d\/, and \/g\/ are all <strong>stops <\/strong>or <strong>plosives<\/strong> \u2013 meaning airflow is entirely blocked by the mouth and nose<\/li>\n<li>[Place of articulation]\n<ul>\n<li>\/b\/ is a <strong>bilabial <\/strong>\u2013 formed with both lips<\/li>\n<li>\/d\/ is an <strong>alveolar <\/strong>\u2013 formed when the tongue tip touches the <strong>alveolar ridge<\/strong> (behind the upper front teeth)<\/li>\n<li>\/g\/ is a <strong>velar <\/strong>\u2013 formed when the tongue body arches towards the <strong>velum<\/strong> or <strong>soft palate<\/strong> (upper back of the mouth)<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>Here\u2019s a little sketch of the vocal tract to help visualize.<\/p>\n<p style=\"text-align: center;\"><strong>Articulators and places of articulation<\/strong><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-254 aligncenter\" src=\"https:\/\/i0.wp.com\/linguamonium.com\/wp-content\/uploads\/2019\/01\/places_of_artic_dwg_alt.jpg?resize=384%2C297&#038;ssl=1\" alt=\"places_of_artic_dwg_alt\" width=\"384\" height=\"297\" data-recalc-dims=\"1\" \/><\/p>\n<p>We can see that the three consonants in <em>ba<\/em>, <em>da<\/em>, and <em>ga<\/em> share voicing and articulatory manner, and only differ in their place of articulation. Interestingly, the \u201cintermediate\u201d consonant that subjects perceived, \/d\/, falls almost <em>exactly <\/em>in-between \/b\/ (the audio consonant) and \/g\/ (the lip-read consonant) in its place of articulation \u2013 \/b\/ is formed at the front of the mouth, \/g\/ is formed at the back, and \/d\/ is formed in the middle. Our brain is combining the conflicting auditory and visual information, and perceiving a sound lying between the two cues. The combination isn\u2019t always this exact, but the perceived phone will usually have some features in common with both sensory inputs.<\/p>\n<p>The McGurk effect is strong and, as far as we know, universal. It happens even when the voice and face are in different locations, or of different genders! All people tested, regardless of their language background, have demonstrated this speech integration behavior. However, the precise <em>kinds<\/em> of synthesis (as well as the effect\u2019s intensity) vary by language. The phenomenon is more apparent in German, English, Spanish, and Turkish language listeners than in Chinese and Japanese participants, for example. The particular syllabic configuration of Japanese, and cultural norms (like face avoidance) may mean that they don\u2019t rely on visual evidence of speech to the same extent. Loud environments heighten the effect for everyone though.<\/p>\n<p>The overall robustness of the McGurk effect indicates just how automatically and unconsciously we combine auditory and visual stimuli when perceiving speech. A slew of related studies demonstrate that our brain really functions on constant multisensory (or <strong>multimodal<\/strong>) data, and is not limited to the sense we take as primary for a particular type of input.<\/p>\n<p>So what other perceptual abilities might be a mix of stuff from several senses? If you have some guesses and are curious to know more, I recommend <em><a href=\"https:\/\/www.amazon.com\/See-What-Im-Saying-Extraordinary\/dp\/0393339378\/ref=sr_1_2?ie=UTF8&amp;qid=1548053785&amp;sr=8-2&amp;keywords=see+what+i'm+saying\" target=\"_blank\" rel=\"noopener\">See What I\u2019m Saying<\/a><\/em>, by Lawrence D. Rosenblum. (In fact, he was the guy in the video above!) Like McGurk and MacDonald, he\u2019s not quite a linguist, but we can forgive him.<\/p>\n<p>&nbsp;<\/p>\n<p><a href=\"#_ftnref1\" name=\"_ftn1\">[1]<\/a> The video says you should have perceived the sound <em>fa <\/em>during the second clip of the man, but I actually get a kind of <em>va <\/em>\/ <em>fa <\/em>mix, which makes sense \u2013 <em>va <\/em>is intermediate in that it\u2019s voiced like <em>ba<\/em> (the audio cue) but has a <strong>labiodental <\/strong>place of articulation, like <em>fa <\/em>(the visual cue).<\/p>\n<p>*Photo attribution: <a href=\"https:\/\/www.flickr.com\/photos\/perpetualplum\/3974880498\" target=\"_blank\" rel=\"noopener\">page 241 Internal Ear<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Think of a situation in which you might try to lip read. Would it be chatting with friends in a noisy bar? Or watching a muted TV? Maybe you were born (or became) deaf. In all of these scenarios, the channel for auditory input is severely obstructed or entirely absent. But what if I told&#8230;<\/p>\n","protected":false},"author":1,"featured_media":257,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_newsletter_tier_id":0,"footnotes":"","jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false}}},"categories":[21,25],"tags":[123,140,141,156,159],"class_list":["post-253","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-phonetics-phonology","category-psycholinguistics","tag-multimodality","tag-perception","tag-phones","tag-speech","tag-studies"],"jetpack_publicize_connections":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v21.7 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Read my lips: McGurk and speech perception - Linguamonium<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/linguamonium.com\/?p=253\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Read my lips: McGurk and speech perception - Linguamonium\" \/>\n<meta property=\"og:description\" content=\"Think of a situation in which you might try to lip read. Would it be chatting with friends in a noisy bar? Or watching a muted TV? Maybe you were born (or became) deaf. In all of these scenarios, the channel for auditory input is severely obstructed or entirely absent. But what if I told...\" \/>\n<meta property=\"og:url\" content=\"https:\/\/linguamonium.com\/?p=253\" \/>\n<meta property=\"og:site_name\" content=\"Linguamonium\" \/>\n<meta property=\"article:published_time\" content=\"2019-01-29T05:03:10+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-04-25T05:34:01+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/linguamonium.com\/wp-content\/uploads\/2019\/01\/internal_ear_cropped.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"497\" \/>\n\t<meta property=\"og:image:height\" content=\"352\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"hannah\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@linguamonium\" \/>\n<meta name=\"twitter:site\" content=\"@linguamonium\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"hannah\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/linguamonium.com\/?p=253#article\",\"isPartOf\":{\"@id\":\"https:\/\/linguamonium.com\/?p=253\"},\"author\":{\"name\":\"hannah\",\"@id\":\"https:\/\/linguamonium.com\/#\/schema\/person\/f6a9c49248cb623f9a6061aaacff0238\"},\"headline\":\"Read my lips: McGurk and speech perception\",\"datePublished\":\"2019-01-29T05:03:10+00:00\",\"dateModified\":\"2021-04-25T05:34:01+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/linguamonium.com\/?p=253\"},\"wordCount\":906,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/linguamonium.com\/#\/schema\/person\/f6a9c49248cb623f9a6061aaacff0238\"},\"keywords\":[\"multimodality\",\"perception\",\"phones\",\"speech\",\"studies\"],\"articleSection\":[\"Phonetics &amp; Phonology\",\"Psycholinguistics\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/linguamonium.com\/?p=253#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/linguamonium.com\/?p=253\",\"url\":\"https:\/\/linguamonium.com\/?p=253\",\"name\":\"Read my lips: McGurk and speech perception - Linguamonium\",\"isPartOf\":{\"@id\":\"https:\/\/linguamonium.com\/#website\"},\"datePublished\":\"2019-01-29T05:03:10+00:00\",\"dateModified\":\"2021-04-25T05:34:01+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/linguamonium.com\/?p=253#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/linguamonium.com\/?p=253\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/linguamonium.com\/?p=253#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/linguamonium.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Read my lips: McGurk and speech perception\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/linguamonium.com\/#website\",\"url\":\"https:\/\/linguamonium.com\/\",\"name\":\"Linguamonium\",\"description\":\"A tumult of linguistic musings\",\"publisher\":{\"@id\":\"https:\/\/linguamonium.com\/#\/schema\/person\/f6a9c49248cb623f9a6061aaacff0238\"},\"alternateName\":\"Linguamo\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/linguamonium.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\/\/linguamonium.com\/#\/schema\/person\/f6a9c49248cb623f9a6061aaacff0238\",\"name\":\"hannah\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/linguamonium.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/linguamonium.com\/wp-content\/uploads\/2018\/06\/cropped-dummeh-babboh-2-1.png\",\"contentUrl\":\"https:\/\/linguamonium.com\/wp-content\/uploads\/2018\/06\/cropped-dummeh-babboh-2-1.png\",\"width\":512,\"height\":512,\"caption\":\"hannah\"},\"logo\":{\"@id\":\"https:\/\/linguamonium.com\/#\/schema\/person\/image\/\"},\"sameAs\":[\"https:\/\/linguamonium.com\",\"www.linkedin.com\/in\/hannah-vanbrunt\",\"https:\/\/twitter.com\/linguamonium\"],\"url\":\"https:\/\/linguamonium.com\/?author=1\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Read my lips: McGurk and speech perception - Linguamonium","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/linguamonium.com\/?p=253","og_locale":"en_US","og_type":"article","og_title":"Read my lips: McGurk and speech perception - Linguamonium","og_description":"Think of a situation in which you might try to lip read. Would it be chatting with friends in a noisy bar? Or watching a muted TV? Maybe you were born (or became) deaf. In all of these scenarios, the channel for auditory input is severely obstructed or entirely absent. But what if I told...","og_url":"https:\/\/linguamonium.com\/?p=253","og_site_name":"Linguamonium","article_published_time":"2019-01-29T05:03:10+00:00","article_modified_time":"2021-04-25T05:34:01+00:00","og_image":[{"width":497,"height":352,"url":"https:\/\/linguamonium.com\/wp-content\/uploads\/2019\/01\/internal_ear_cropped.jpg","type":"image\/jpeg"}],"author":"hannah","twitter_card":"summary_large_image","twitter_creator":"@linguamonium","twitter_site":"@linguamonium","twitter_misc":{"Written by":"hannah","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/linguamonium.com\/?p=253#article","isPartOf":{"@id":"https:\/\/linguamonium.com\/?p=253"},"author":{"name":"hannah","@id":"https:\/\/linguamonium.com\/#\/schema\/person\/f6a9c49248cb623f9a6061aaacff0238"},"headline":"Read my lips: McGurk and speech perception","datePublished":"2019-01-29T05:03:10+00:00","dateModified":"2021-04-25T05:34:01+00:00","mainEntityOfPage":{"@id":"https:\/\/linguamonium.com\/?p=253"},"wordCount":906,"commentCount":0,"publisher":{"@id":"https:\/\/linguamonium.com\/#\/schema\/person\/f6a9c49248cb623f9a6061aaacff0238"},"keywords":["multimodality","perception","phones","speech","studies"],"articleSection":["Phonetics &amp; Phonology","Psycholinguistics"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/linguamonium.com\/?p=253#respond"]}]},{"@type":"WebPage","@id":"https:\/\/linguamonium.com\/?p=253","url":"https:\/\/linguamonium.com\/?p=253","name":"Read my lips: McGurk and speech perception - Linguamonium","isPartOf":{"@id":"https:\/\/linguamonium.com\/#website"},"datePublished":"2019-01-29T05:03:10+00:00","dateModified":"2021-04-25T05:34:01+00:00","breadcrumb":{"@id":"https:\/\/linguamonium.com\/?p=253#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/linguamonium.com\/?p=253"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/linguamonium.com\/?p=253#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/linguamonium.com\/"},{"@type":"ListItem","position":2,"name":"Read my lips: McGurk and speech perception"}]},{"@type":"WebSite","@id":"https:\/\/linguamonium.com\/#website","url":"https:\/\/linguamonium.com\/","name":"Linguamonium","description":"A tumult of linguistic musings","publisher":{"@id":"https:\/\/linguamonium.com\/#\/schema\/person\/f6a9c49248cb623f9a6061aaacff0238"},"alternateName":"Linguamo","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/linguamonium.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":["Person","Organization"],"@id":"https:\/\/linguamonium.com\/#\/schema\/person\/f6a9c49248cb623f9a6061aaacff0238","name":"hannah","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/linguamonium.com\/#\/schema\/person\/image\/","url":"https:\/\/linguamonium.com\/wp-content\/uploads\/2018\/06\/cropped-dummeh-babboh-2-1.png","contentUrl":"https:\/\/linguamonium.com\/wp-content\/uploads\/2018\/06\/cropped-dummeh-babboh-2-1.png","width":512,"height":512,"caption":"hannah"},"logo":{"@id":"https:\/\/linguamonium.com\/#\/schema\/person\/image\/"},"sameAs":["https:\/\/linguamonium.com","www.linkedin.com\/in\/hannah-vanbrunt","https:\/\/twitter.com\/linguamonium"],"url":"https:\/\/linguamonium.com\/?author=1"}]}},"jetpack_featured_media_url":"https:\/\/i0.wp.com\/linguamonium.com\/wp-content\/uploads\/2019\/01\/internal_ear_cropped.jpg?fit=497%2C352&ssl=1","jetpack_likes_enabled":true,"jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/linguamonium.com\/index.php?rest_route=\/wp\/v2\/posts\/253","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/linguamonium.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/linguamonium.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/linguamonium.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/linguamonium.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=253"}],"version-history":[{"count":2,"href":"https:\/\/linguamonium.com\/index.php?rest_route=\/wp\/v2\/posts\/253\/revisions"}],"predecessor-version":[{"id":518,"href":"https:\/\/linguamonium.com\/index.php?rest_route=\/wp\/v2\/posts\/253\/revisions\/518"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/linguamonium.com\/index.php?rest_route=\/wp\/v2\/media\/257"}],"wp:attachment":[{"href":"https:\/\/linguamonium.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=253"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/linguamonium.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=253"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/linguamonium.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=253"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}