Brain Advance Access published online on September 29, 2007
Brain, doi:10.1093/brain/awm230
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Visual speech circuits in profound acquired deafness: a possible role for latent multimodal connectivity
1Inserm U742 F-75005 Paris, Université Pierre et Marie Curie (Paris-6) F-75005 Paris & Département d’Etudes Cognitives, Ecole Normale Supérieure, F-75005 Paris, France, 2Department of Otolaryngology, Hallym University College of Medicine, Anyang, Korea, 3Service d’ORL, Hôpital Edouard Herriot and 4CERMEP Imagerie du vivant, F-69003 Lyon, France
Correspondence to:
Dr Anne-Lise Giraud, Département d’Etudes Cognitives – Ecole Normale Supérieure, 29, rue d’Ulm – 75005 Paris, France E-mail: anne-lise.giraud{at}ens.fr
 |
Summary |
|---|
 Top Summary IntroductionMethodsResultsDiscussionConclusionsSupplementary MaterialReferences |
|---|
It is commonly held that losing one sense provokes cross-modal takeover of deprived cortical areas, and therefore results in a benefit for the remaining modalities. Using functional magnetic resonance imaging (fMRI), we assessed the impact of acquired deafness on the brain network related to speechreading and teased apart cortical areas with responses showing long-term reorganization, i.e. time-dependent plasticity over 4–48 months of deafness, from those expressing compensation, i.e. performance-related activity. Nine deaf patients (7 women, age; mean ± SE. = 50.2 ± 4.8) and control subjects performed equally well in a visual speechreading task but deaf patients activated the left posterior superior temporal cortex more than controls. This effect correlated with speechreading fluency but not with the duration of sensory deprivation, thus arguing against long-term reorganization as the source of these cross-modal effects. To the contrary, cross-modal activation in the left posterior superior temporal cortex of deaf patients decreased with deafness duration. Our observation that cross-modal effects were most pronounced right after deafness onset is at odds with the classical view on brain reorganization. We suggest that functional compensation of sensory deprivation does not require slowly progressive colonization of superior temporal regions by visual inputs, but can exploit a switch to pre-existing latent multimodal connectivity.
Key Words: deafness; speechreading; fMRI; multimodal; reorganization
Abbreviations: fMRI, functional magnetic resonance imaging; PSNHL, progressive sensorineural hearing loss; STG, superior temporal gyrus; STS, superior temporal sulcus
Received May 11, 2007. Revised August 23, 2007. Accepted August 24, 2007.
|
Introduction |
|---|
|
TopSummaryIntroductionMethodsResultsDiscussionConclusionsSupplementary MaterialReferences |
|---|
Thanks to multisensory competition for cortical space, profound deafness provokes brain reorganization that benefits the remaining senses (Levanen and Hamdorf, 2001
). Visual capacities in profound congenitally deaf patients, however, appear only enhanced in behavioural domains that would normally benefit from audio-visual integration, and that are under strong attentional top–down control (Bavelier et al., 2006). Two neural mechanisms may account for these behavioural effects. The loss of auditory input in direct projections from auditory to visual areas (Falchier et al., 2002) could possibly enhance susceptibility of visual areas to visual inputs and visual top–down control (Bavelier et al., 2006). Alternatively, the temporal cortex may undergo cross-modal reorganization so that visual stimuli tap into regions that normally respond predominantly to sounds (Calvert et al., 1997; Petitto et al., 2000; Bavelier et al., 2001; Finney et al., 2003; Fine et al., 2005; Pekkola et al., 2006). Accordingly, there is abundant evidence for visual (or tactile) responses in temporal cortex of the deaf, even reaching the primary auditory region (Puce et al., 1998; MacSweeney et al., 2002, 2004; Finney et al., 2003; Fine et al., 2005; Lambertz et al., 2005; Doucet et al., 2006).
Visual responses in superior temporal cortex could result from long-term cross-modal reorganization, but they might also reveal more dynamic functional phenomena. In acquired deafness in particular, cross-modal responses might indicate an immediate change in the relative contribution of auditory/visual/somatosensory input to the temporal cortex. This view implies that superior temporal regions that are classically held as unimodal auditory, are covertly hetero- or multimodal (Beauchamp, 2005; Ghazanfar and Schroeder, 2006; von Kriegstein and Giraud, 2006). This property may result from converging inputs to the temporal region from different sensory cortices (Ghazanfar et al., 2005), or from low-level functional interactions across associative cortices creating multimodal functional circuits (Beauchamp, 2005; von Kriegstein and Giraud, 2006). These circuits have been shown to become rapidly functional once sensory or task requirements change (von Kriegstein and Giraud, 2006). It is therefore possible that auditory deprivation, rather than inducing slow progressive structural reorganization of temporal cortex, provokes a rapid functional reorganization of latent multimodal circuits, e.g. by disinhibiting far-reaching connectivity with other sensory modalities.
In this study, we pursued the hypothesis that, if cross-modal responses following sensory deprivation result from temporal reorganization relying on new connectivity and synaptogenesis, visual responses in temporal cortex in deaf people should build-up over months to years following sensory deprivation. If, conversely, cross-modal responses result from immediate reorganization, e.g. a takeover of latent multimodal connectivity, visual responses in temporal cortex should be observed rapidly, irrespective of the time elapsed since deafness onset.
This dichotomy has important implications with respect to the behavioural benefit that cross-modal responses may provide. If latent networks are unveiled by auditory deprivation, their participation in global functional network should moderately be disturbed, and they should readily participate in effective compensatory mechanisms, whereas if rewiring and synaptogenesis underpin cross-modal responses, related behavioural effects should build-up over months to years after deafness onset (Giraud et al., 2001; Smirnakis et al., 2005).
Speechreading is the major communication mode in acquired profound deafness, and should thus be under strong reorganization demands. By disentangling neural activity related to deafness duration from that related to speechreading fluency during a speechreading task, we assessed the contribution of long-term versus short-term plasticity to the shift towards exclusive visual communication in deafened adults.
|
Methods |
|---|
|
TopSummaryIntroductionMethodsResultsDiscussionConclusionsSupplementary MaterialReferences |
|---|
Participants
Twelve post-lingually deaf adults without known neurological disease were enrolled in a research investigation that was approved by the local ethics committee (CPP, Sud-Est IV, Centre Léon Bérard, Lyon, France). Ten of them had bilateral sensorineural hearing loss since childhood yielding profound deafness in adulthood. The other two patients were deafened by ototoxic drug and head trauma. None of them used sign language, and they all relied on speechreading and reading for communication.
Three deaf patients were excluded from the functional magnetic resonance imaging (fMRI) analysis because they could not perform the tests correctly during scanning, leaving data from 9 deaf patients 7 women and 2 men, age; mean ± SE = 50.2 ± 4.8, see Table 1 for demographic data) and 15 controls (7 men and 8 women, age; mean ± SE = 32.2 ± 3.4). To minimize age effect in group comparisons, the latter were performed only with the 9 oldest control subjects (3 men and 6 women, age; mean ± SE = 39.3 ± 4.2). Data from the other six controls, however, were included in a conjunction analysis that assessed effects that were consistently present in deaf subjects and absent in controls. All controls had normal hearing and no audiological or neurological antecedents. Written informed consent was obtained from all participants. All of them had normal or corrected-to-normal visual acuity, and were right-handed according to a modified version of Edinburgh Inventory of Handedness (Oldfield, 1971).
|
Assessment of clinical parameters
In deaf patients, speechreading fluency was assessed before the fMRI investigation by a standard French sentence test with visual cue only. A list composed of 100 three-phoneme words (Lafon, 1964
) was presented by a speech therapist and the correctly identified words were scored. Duration of deafness was determined as the number of months elapsed since onset of profound hearing loss. Onset of profound deafness is defined as the time point when oral communication was fully compromised even with the best-fitted hearing aid. Clinical details are provided in Table 1 for the 9 deaf subjects who participated in the fMRI data analysis.
FMRI experiment
Stimuli
During fMRI, subjects performed a speechreading task (SR) from short videos of a French native male pronouncing French numbers from 1 to 9. In the original version of the videos, the speaker pronounced the numbers aloud and the sound track was subsequently removed. In a baseline condition (CO), the same speaker made closed-mouth gurning movements at the same rate as the pronounced numbers. The stimuli were edited by iMovie version 4.0.1 on Macintosh (Apple computer, Inc., USA). Each video lasted 2 s. Duration, onset time and magnitude of mouth movements were equated across conditions.
Experimental protocol and design
To ensure that all participants had understood the instructions correctly and could identify numbers by speechreading, subjects were trained with a computer program using the same stimuli as those used during scanning. After training, all participants showed over 80% correct identification rate during the speechreading task. During fMRI, subjects either covertly repeated the number pronounced by the speaker (SR) or counted the number of face videos (always the same person) that appeared on the screen (CO). In both tasks, subjects pressed a response button with the right index finger whenever the identified number or the count was even. The two tasks were thus overall balanced with respect to motor output as even numbers occurred as often as even counts. The internally represented linguistic content and the task-related semantic judgement were also balanced across conditions in this way. Videos were presented randomly to prevent anticipation and force speechreading in the SR condition. In the control task however, randomization of gurning faces did not determine the output and subjects counted regularly from 1 to 9. Hence, there remained an inevitable difference between conditions in predictability of content and task response.
The fMRI experiment comprised two runs, with five 2-min SR blocks and five CO blocks, randomly presented using Presentation software version 9.90 (Neurobehavioral system, Inc., Albany, CA, USA). This experiment was part of a larger protocol designed to test various aspects of visual language in deaf people (see Supplementary Material). Each block began with a screen showing instructions for 2 s, and 9 stimuli followed in random order with a 1 s black screen in between stimuli. A fixation cross was presented at the beginning and the end of each block and a 25 s black screen followed each block.
The stimuli were back-projected onto a screen positioned at the base of the scanner by a Canon Xeed SX50 LCD projector, and viewed via a mirror angled above the subject's head in the scanner.
Imaging parameters
Gradient echo-planar MRI data were acquired with a 1.5T magnetic resonance scanner (Siemens Sonata) with standard head coil to obtain volume series with 27 contiguous transverse oblique slices of 4 mm thickness (voxel size 3.4 x 3.4 mm2 in transverse plane, no gap, TR 2.6 s, TE 60 ms) covering the whole brain. Earplugs and earmuffs were provided both to controls and deaf subjects to equate experimental environment. In each run, 244 functional images were acquired. The first four images from each run were discarded to reach a steady state—each experimental session thus yielded 480 scans per subject.
Data analysis
We analysed the fMRI data using SPM2 (Wellcome Department of Cognitive Neurology, London, UK, http://www.fil.ion.ucl.ac.uk/spm) in a Matlab 7.1 (Mathworks, Sherborn, MA, USA) environment. Standard preprocessing procedures (realignment and unwarping, normalization and spatial smoothing with an 8-mm full width at half-maximum Gaussian kernel) were performed and statistical block-design analyses including two conditions were modelled for individual data analysis. The output of these individual analyses was entered into three second-level, random-effect (RFX) analyses.
Group analyses: main effects and group differences
An ANOVA with two groups per condition examined the main effects of condition in each group. We report, in Table 2, the effects that are significant at P = 0.05, FWE-corrected for multiple comparisons. We used a statistical threshold of P = 0.001 (T = 3.37), for group comparisons (Table 3).
|
|
Speechreading effects common to all deaf subjects relative to controls
We ran another ANOVA including contrast images of the speechreading condition from each of the 9 deaf subjects and all 15 hearing controls, and performed a conjunction analysis (deaf 1 > controls and deaf 2 > controls up to deaf 9 > controls) to identify regions that were consistently overactivated in every deaf patient compared to the control group.
Multiple regression analysis of deafness duration and speechreading performance factors
Effects due to duration of deafness and lipreading fluency in deaf patients were assessed in the fMRI dataset. Contrast images of the speechreading condition in deaf patients were entered into a multiple regression analysis with the two clinical factors as covariates to identify brain regions correlating with each factor. Age was additionally entered in this analysis to remove its possible influence on the experimental variance. In a further step, findings on speechreading effects from the deaf patients’ group were used to define an inclusive mask (corrected P = 0.05), and the results were thresholded using a voxel height threshold of P < 0.01 (T = 3.14), uncorrected.
|
Results |
|---|
|
TopSummaryIntroductionMethodsResultsDiscussionConclusionsSupplementary MaterialReferences |
|---|
Clinical measures
We first analysed how deafness impacted on the performance of visual speech by assessing the correlation between deafness duration and lipreading fluency measured during an open set clinical test. In the group of 12 subjects, the two factors appeared independent of each another (Fig. 1, P = 0.167, correlation coefficient r = –0.431), indicating that those subjects with longer deafness duration were not the ones with the best speechreading scores. Although all deaf patients had higher scores in our closed set task, their performance in open set showed a larger variance (40–85% among those patients enrolled in the fMRI protocol).
|
Behavioural effects in deafs and controls
We analysed the fMRI dataset to compare the neural circuits involved in speechreading in post-lingual deaf and normal-hearing subjects, relative to counting the occurrence of gurning faces, a task that controlled for semantic and most of the executive aspects of the speechreading task. By using a closed set of stimuli, we ensured that the task was easy to perform in both groups, and that it entailed no advantage for the deaf group. Accordingly, both groups obtained over 90% correct rate on target trials in both conditions (Fig. 2) and performance showed no statistical difference across groups (Mann–Whitney U-test, P = 0.345/0.753 for percent correct trials and P = 0.294/0.401 for response time RT in speechreading/counting conditions). However, accuracy was lower and RT longer during speechreading in controls than during counting in both groups (Wilcoxon signed rank test, P = 0.043/0.106 for percent correct trials and P = 0.017/0.012 for RT in deaf patients/hearing controls). This difference across tasks reflects the difference between a predictable (counting) versus a stimulus-dependent (speechreading) motor output.
|
Neural activity in deaf patients—cross-modal effects
During speechreading relative to rest, patients and controls activated common areas including bilateral visual cortex, fusiform, middle temporal, angular gyrus, the insula and the lateral prefrontal cortex (Fig. 3), in keeping with previous data (Calvert et al., 1997
; Calvert and Campbell, 2003; Hall et al., 2005). When counting gurning faces, activation levels throughout this network dropped in controls but remained elevated in patients (Fig. 3, Table 2), revealing increased automaticity in dynamic face processing, irrespective of whether faces express meaningful speech or not. Enhanced automaticity in deaf patients relative to controls also showed through an increase in basal ganglia (putamen) activity during both tasks (Booth et al., 2007). All moving faces activated regions that classically process auditory speech stimuli (Scott and Johnsrude, 2003), which confirms that auditory deprivation entails reorganization. In addition, we observed a significant task-by-group interaction in the left posterior superior temporal cortex, indicating that speechreading activated this region more than viewing gurning faces, in patients relative to controls (Fig. 4, green spot).
|
|
Directly contrasting brain responses between deafs and controls in the speechreading condition confirmed that many brain regions shifted their main input modality, i.e. they express a cross-modal effect. Deaf patients had higher activity levels in all auditory association regions including bilateral superior temporal gyrus (STG) and superior temporal sulcus (STS), anterior temporal, visual, ventral prefrontal cortices and bilateral temporoparietal junctions, extending to the inferior parietal lobule on the left side (Fig. 4, Table 3). The left posterior superior and the right anterior temporal cortices were activated more during speechreading than gurning (Fig. 4a and e), but only the left posterior superior temporal cortex (–40, –38, 12) was consistently more active in each single deaf patient relative to the hearing controls (Fig. 4, red spot, and 4b). It was at this location that we also observed a significant group-by-task interaction. Visual cortices (Fig. 4d), bilateral angular gyrus (Fig. 4c, activation profile was similar in both angular gyri) and right prefrontal regions (Fig. 4f) were overactivated in patients during both speechreading numbers and counting gurning faces, indicating that these regions were not directly relevant to speech communication after deafness, but possibly contributed to enhance globally the visual processing of speech sources, i.e. faces (Womelsdorf et al., 2006
; Moradi et al., 2007). In contrast, hearing participants had higher activations than deaf patients in medial and lateral frontopolar regions (BA46 and BA10), but no interaction with task was observed.
Multifactorial analysis
Regions where the activity level during speechreading varied as a function of (i) deafness duration, and (ii) speechreading fluency measured prior to the fMRI experiment were assessed in a multivariate analysis. The possible influence of age was discarded by including age as a nuisance variable in the analysis. Positive correlation with speechreading fluency was found in bilateral dorsal and ventral prefrontal cortex, and bilateral insula, in bilateral posterior superior temporal cortex with a left predominance, bilateral STS and bilateral visual association areas (with a left predominance), and in the right fusiform region (40 – 52 – 14) (Fig. 5). Speechreading activity negatively correlated with deafness duration in bilateral posterior superior temporal regions and visual cortices, in overlap with regions where activity reflected speechreading scores. In addition, deafness duration correlated negatively with activity in the left angular gyrus, the left prefrontal (Broca's area) and bilateral fusiform cortices. Positive correlation with deafness duration was limited to restricted parts of both prefrontal cortices, and the right temporoparietal junction.
|
As our intention was to assess the contribution of deafness duration and its possible interaction with performance for regions that are cross-modally activated after deafness, we focus on the examination of the multivariate analysis to the subset of regions that were overactivated in deaf patients relative to controls (the yellow network in Fig. 4). In the region of left posterior STG that showed speech-specific effects, we observed a positive correlation with speechreading fluency and a negative correlation with deafness duration (Fig. 5d) that extended posteriorly (Fig. 5e). The same profile was observed in the visual cortex (Fig. 5f), and in the right posterior STG (Fig. 5c, note that the latter was not more activated in patients than controls). This profile hence dominated across the cross-modal network; whereas only the right angular gyrus and Broca's area showed a positive correlation with deafness duration (Fig. 5a and b). Activity in the latter region also correlated positively with speechreading fluency, which suggests progressive compensatory mechanisms that improve visual speech processing. For the angular gyrus, however, the correlation of activity with deafness duration did not appear to be behaviourally relevant as there was no associated correlation with speechreading fluency.
|
Discussion |
|---|
|
TopSummaryIntroductionMethodsResultsDiscussionConclusionsSupplementary MaterialReferences |
|---|
Cross-modal reorganization in speech regions
Relative to normal-hearing subjects, deaf patients exhibited enhanced responses to dynamic faces in left STG/STS speech regions. In deaf patients, regions that normally receive predominant auditory input have thus reorganized to process visual input that potentially contains speech information. Neural responses to speechreading in these regions correlated with individual speechreading fluency scores, which confirm previous observations in normal-hearing subjects (Ludman et al., 2000
; Hall et al., 2005). In these same regions, we did not detect a positive correlation with deafness duration between 4 and 48 months of sensory deprivation. Even though speechreading does not usually improve with time and effort (Andersson et al., 2001a), it could be argued that the strong cognitive and motivational components attached to speechreading in profound deafness should enhance neural plasticity (Blake et al., 2006). We take this negative result to indicate that visual responses in auditory speech regions do not reflect progressive takeover of auditory regions by visual input. This interpretation is in line with recent animal studies of adult brain plasticity showing only limited reorganization in visual regions that were deprived of their main sensory input within a time span similar to the one we investigated here (Smirnakis et al., 2005). Our data rather suggest that cross-modal reorganization in speech regions was already present 4 months after deafness onset, which speaks for changes occurring close to deafness onset. Our findings additionally show that these changes are immediately relevant for human communication.
In keeping with our observations, rapid reorganization following deprivation is observed in the motor cortex of hand amputees (Reilly et al., 2006). In contrast with the classical view that other representations progressively colonize deprived cortical area, EMG response profiles during phantom hand movements (Reilly et al., 2006) indicate that the hand representation immediately utilizes the arm motor territory to express itself, thanks to pre-existing overlapping/redundant representations. In the domain of audio-visual speech processing, this novel take on cortical plasticity suggests that what we used to interpret as cross-modal reorganization after deafness reflects the opposite tendency of cerebral dynamics, i.e. the resilience of audio-visual speech networks to sensory perturbations. Only the maintenance of stable speech representations, and their access through partially redundant audio-visual connectivity could potentially preserve communication in acquired deafness.
Plasticity of the audio-visual speech network
Audio-visual speech networks are established early in life, and rapidly take an irreversible configuration. Schorr et al. (2005) showed that children born deaf and fitted with cochlear implants are susceptible to the McGurk effect if auditory restoration occurs before the age of 30 months. After this age, their perception of audio-visual syllables is visually biased. These findings speak for a sensitive period in audio-visual speech, after which auditory input can no longer be fused with visual input, with the positive consequence that the established network becomes resilient to subsequent sensory damage. In adults, speechreading performance varies widely across individuals, and neither explicit training nor visual communication demands appear to greatly enhance performance (Bernstein et al., 2000, 2001, Andersson et al., 2001b; Summerfield, 1992). Given that speechreading ability is grounded in audio-visual binding, and that visual phonological forms are ambiguous by nature, the possibilities to improve speechreading in a unimodal visual setting are limited. Disambiguation of visual phonological forms thus requires additional sensory information, i.e. added hand gestures as in Cued Speech (Charlier and Leybaert, 2000).
If deafness occurs after such a critical period for learning audio-visual associations, phonological memory will progressively deteriorate (Lyxell et al., 1994). This process is slow and the effect subtle, thus preserving the capacity for re-learning audio-visual matching if hearing is subsequently restored by cochlear implant. The more intact the pre-established audio-visual phonological network remains during deafness, the more immediate the efficiency of oral speech comprehension in case of reafferentation. Previous functional neuroimaging data in cochlear implantees support this idea by showing that progressive increase in auditory and visual responses to speech over years after auditory restoration parallels an improvement in speechreading performance (Tyler et al., 1995; Giraud et al., 2001). This effect was observed in good implant users only, but not in a larger cohort including patients with all possible speech scores (van Dijk et al., 1999; Rouger et al., 2007). This suggests that either deaf patients with poor auditory speech scores had already very high speechreading skills, which thus cannot further improve after restoration of audio-visual matching, or more likely, that those patients with poor auditory gain had poor phonological representations and poor speechreading performance before deafness, limiting a possible improvement of audio-visual phonological matching and speechreading skills after implantation. It is also probable that those hearing subjects with poor audio-visual fusion are likely to become poor speechreaders in case of acquired deafness. At any rate, the dynamics of reorganization in speech regions that we observed here, is compatible with the idea that reactivity to sensory impairment is both determined and constrained by the latent connectivity that has developed during childhood, and has (moderately) been tuned throughout life.
Underlying audio-visual connectivity of speech regions at the time of deafness
Cross-modal responses to visual speech are underpinned by a robust audio-visual network (Campbell, 1998). Perisylvian temporal regions respond to both auditory and visual inputs, with predominant auditory responses (MacSweeney et al., 2004), indicating that neurons in these regions are at least bimodal. The early effects we detect here as increased responses relative to controls and a negative correlation with deafness duration most probably reflect and express the effective connectivity that is structurally present at the moment of deafness. Eight of the 9 patients enrolled in the fMRI study became deaf due to progressive sensorineural hearing loss (PSNHL). This condition could have slowly modified the audio-visual balance in speech communication. Our results indicate that speech regions express their underlying potential to process visual speech right at the onset of deafness. That this potential has been tuned by increased reliance on vision in the context of PSNHL is possible, and would agree with the principle of inverse effectiveness, which presupposes that multimodal compensation is maximal when one sense is the least effective (Meredith and Stein, 1996). That the only patient with abrupt hearing loss had a lower speechreading score than the other patients with the same deafness duration is compatible with this view (circled data in Fig. 5). To test this hypothesis, we correlated speechreading responses with the duration of progressive hearing loss. However, we did not detect any statistically significant effect, and no subthreshold effect overlapped with the left posterior STG/STS region. It is to be noted that the assessment of the onset of progressive hearing loss is rather unreliable. Invalidation of this hypothesis would therefore also require studying speechreading in PSNHL at various stages preceding deafness. Our data do not prove an explicit effect of PSNHL, although this type of aetiology is likely to play a role. They rather point to the resilience of the audio-visual speech network to perturbation of sensory balance. Our findings show that over and above any potential effect of PSNHL, cross-modal reorganisation of speech regions occurs rapidly once auditory input is no longer available, without progressing towards visual colonization of auditory speech regions.
What type of reorganization?
Audio-visual speech processing is underpinned by two possible connectivity principles. The classical view is that auditory and visual inputs converge on the same neural substrate. This indeed could account for many fusion effects, e.g. that visual information from speaker's face enhances speech intelligibility even though lipreading is not necessary for speech comprehension, (Erber, 1969; Summerfield, 1979; MacLeod and Summerfield, 1987; van Wassenhove et al., 2005). This convergence principle, however, lacks the flexibility to account for the more general observation that new multimodal combinations can become rapidly effective even in adults (Sheffert and Olson, 2004; Lehmann and Murray, 2005). Flexible multimodal binding could operate via small networks of distributed sensory representations (Beauchamp, 2005). Thus, even sparse unimodal sensory input, as in speechreading, could tap into a broad range of sensory associations (von Kriegstein et al., 2005, 2006; von Kriegstein and Giraud, 2006). This associative principle would (i) enhance the overall detectability of unimodal events by engaging more cortical substrate, and (ii) increase speed and accuracy of unimodal stimulus identification by eliciting specific predictive loops between association areas.
Regarding plasticity, the distinction between associative loops and association by convergent inputs is important, as reorganization of inputs to a common region should continuously evolve towards complete visual dominance in the absence of competitive auditory input, while reorganization of associative loops would lose efficiency with time in the absence of mutual reinforcement from each input modality. By showing a negative correlation with deafness duration in speech regions, our results are compatible with reorganization of associative loops between auditory and visual regions. These results align with others indicating that efficient functional rearrangements are very likely to occur immediately after sensory deprivation (days to months) in regions displaying complex and far-reaching connectivity, such as speech regions, and that they reflect a reorganization within pre-existing connectivity patterns rather than synaptic rearrangements (Smirnakis, 2005).
Phonological processing after deafness
Although post-lingually deaf patients critically rely on lipreading in everyday communication, our results confirmed that speechreading skills saturate. They do not improve, or even slightly deteriorate over time (Lyxell et al., 1994), which is congruent with the intuition that, in the absence of feedback from speech input, phonological representations progressively fade out (Andersson and Lyxell, 1998). In our fMRI dataset, phonological deterioration could account for the negative correlation of deafness duration with activity in the left posterior temporal cortex (Fig. 5d and e), and in dorsal Broca's area (–52 6 44 P = 0.001; Gold and Buckner, 2002). The largest effect in phonological regions was thus observed in those patients who deafened most recently, which underlines the potential of these regions to rapidly readjust their receptive properties. Moreover, it also appears that long-term effects counteract the potential of these regions to access phonological representations from visual inputs. This observation contradicts the classical view that visual inputs should progressively invade regions that are normally under strong auditory influence.
Further findings (beyond the scope of this report) demonstrated a specific deterioration of phonological processing with time elapsed since deafness onset. In 8 of the 9 patients enrolled in the speechreading fMRI experiment, we performed an additional fMRI experiment involving a visual phonological memory task (Supplementary Material). Contrary to speechreading, this visual phonological task did not elicit higher left posterior STG responses in patients than in controls, but a negative correlation between phonology-related neural activity and deafness duration was observed in this region (Wernicke's area and left supramarginal gyrus, Supplementary Fig.; Jacquemot et al., 2003; Vigneau et al., 2006). This effect was associated with increased reaction times and increased activity in other brain regions, which further support that deaf patients progressively lose efficiency in phonological processing and attempt to compensate otherwise.
Long-term global compensatory effects
Together, our results show that after onset of deafness, auditory speech regions rapidly enhance their receptiveness to visual inputs. With progressive duration of deafness, the phonological processing carried out by these regions becomes less and less accurate. Long-term compensation mechanisms might partly counteract degraded phonological representations by enhancing active phonological search and attentional control.
The long-term effects that we detected as positive correlation with deprivation duration are compatible with such compensatory cognitive strategies. These effects were limited and not lateralized to the left hemisphere although the task targeted language. Time-dependent effects were detected in bilateral prefrontal cortex and in the right angular gyrus. Activity in bilateral inferior prefrontal regions correlated positively with both deafness duration and speechreading fluency, and the right angular gyrus with deafness duration only. The prefrontal cortex integrates inputs from several sensory modalities and controls behaviour (Duncan and Owen, 2000; Koechlin et al., 2003), and the right inferior parietal region is involved in spatial attention (Hillis et al., 2005) and action awareness (Farrer et al., 2007). Both regions are likely to mediate conscious compensatory strategies to palliate a progressive phonological deficit. Yet, activity in the right angular gyrus did not correlate with speechreading fluency, and was equal for speechreading and gurning faces (Fig. 4c). This region might thus govern aspects of the speechreading task that did not directly determine speechreading performance, possibly through global visual task monitoring.
Interestingly, activity levels in these prefrontal and parietal regions increased linearly within the 4–48 months time window of our study without reaching a plateau effect. In other words, these effects had not yet saturated 4 years after deafness onset, and may thus express reorganization for instance via synaptogenesis or other slow but long-term plastic mechanisms (Baker et al., 2005).
|
Conclusions |
|---|
|
TopSummaryIntroductionMethodsResultsDiscussionConclusionsSupplementary MaterialReferences |
|---|
Comparison of functional speechreading circuits in hearing controls and deaf subjects delineated brain regions that cross-modally reorganize after auditory deprivation. We did not observe an increase in neural activity in these regions between 4 and 48 months after deafness onset. On the contrary, speech regions showed a progressive decrease in activity with the time elapsed since deafness, and their activity correlated positively but independently with speechreading fluency scores. We take these results to indicate that auditorily deprived speech regions do not undergo progressive cross-modal reorganization, but rather express their underlying potential to respond to visual speech right at the onset of deafness. We propose that cross-modal reorganization relevant to speechreading reflects the configuration of latent multimodal circuits that have been tuned by multisensory experience preceding deafness.
|
Supplementary Material |
|---|
|
TopSummaryIntroductionMethodsResultsDiscussionConclusionsSupplementary MaterialReferences |
|---|
Supplementary material is available at Brain online.
|
Acknowledgements |
|---|
The authors are thankful to Benjamin Morillon for his contribution in stimulus preparation, Michael Gaebler for analysing supplemental data, and Richard Frackowiak and Andreas Kleinschmidt for correcting the manuscript. This research was conducted with financial support from Fyssen Foundation, Collin ORL-France, Cochlear-France, Fondation pour l’Avenir (Grant ET6-436).
|
References |
|---|
|
TopSummaryIntroductionMethodsResultsDiscussionConclusionsSupplementary MaterialReferences |
|---|
Andersson U, Lyxell B. Phonological deterioration in adults with an acquired severe hearing impairment. Scand Audiol Suppl (1998) 49:93–100.[Medline]
Andersson U, Lyxell B, Ronnberg J, Spens KE. Cognitive correlates of visual speech understanding in hearing-impaired individuals. J Deaf Stud Deaf Educ (2001a) 6:103–16.
Andersson U, Lyxell B, Ronnberg J, Spens KE. Effects of tactile training on visual speechreading: performance changes related to individual differences in cognitive skills. J Deaf Stud Deaf Educ (2001b) 6:116–29.
Baker CI, Peli E, Knouf N, Kanwisher NG. Reorganization of visual processing in macular degeneration. J Neurosci (2005) 25:614–8.
Bavelier D, Brozinsky C, Tomann A, Mitchell T, Neville H, Liu G. Impact of early deafness and early exposure to sign language on the cerebral organization for motion processing. J Neurosci (2001) 21:8931–42.
Bavelier D, Dye MW, Hauser PC. Do deaf individuals see better? Trends Cogn Sci (2006) 10:512–8.[CrossRef][Web of Science][Medline]
Beauchamp MS. See me, hear me, touch me: multisensory integration in lateral occipital-temporal cortex. Curr Opin Neurobiol (2005) 15:145–53.[CrossRef][Web of Science][Medline]
Bernstein LE, Auer ET Jr, Tucker PE. Enhanced speechreading in deaf adults: can short-term training/practice close the gap for hearing adults? J Speech Lang Hear Res (2001) 44:5–18.
Bernstein LE, Demorest ME, Tucker PE. Speech perception without hearing. Percept Psychophys (2000) 62:233–52.[Web of Science][Medline]
Blake DT, Heiser MA, Caywood M, Merzenich MM. Experience-dependent adult cortical plasticity requires cognitive association between sensation and reward. Neuron (2006) 52:371–81.[CrossRef][Web of Science][Medline]
Booth JR, Wood L, Lu D, Houk JC, Bitan T. The role of the basal ganglia and cerebellum in language processing. Brain Res (2007) 1133:136–44.[CrossRef][Web of Science][Medline]
Calvert GA, Bullmore ET, Brammer MJ, Campbell R, Williams SC, McGuire PK, et al. Activation of auditory cortex during silent lipreading. Science (1997) 276:593–6.
Calvert GA, Campbell R. Reading speech from still and moving faces: the neural substrates of visible speech. J Cogn Neurosci (2003) 15:57–70.[CrossRef][Web of Science][Medline]
Campbell R. Speechreading: advances in understanding its cortical bases and implications for deafness and speech rehabilitation. Scand Audiol Suppl (1998) 49:80–6.[Medline]
Charlier BL, Leybaert J. The rhyming skills of deaf children educated with phonetically augmented speechreading. Q J Exp Psychol A (2000) 53:349–75.[CrossRef][Web of Science][Medline]
Doucet ME, Bergeron F, Lassonde M, Ferron P, Lepore F. Cross-modal reorganization and speech perception in cochlear implant users. Brain (2006) 129:3376–83.
Duncan J, Owen AM. Common regions of the human frontal lobe recruited by diverse cognitive demands. Trends Neurosci (2000) 23:475–83.[CrossRef][Web of Science][Medline]
Erber NP. Interaction of audition and vision in the recognition of oral speech stimuli. J Speech Hear Res (1969) 12:423–5.[Web of Science][Medline]
Falchier A, Clavagnier S, Barone P, Kennedy H. Anatomical evidence of multimodal integration in primate striate cortex. J Neurosci (2002) 22:5749–59.
Farrer C, Frey SH, Van Horn JD, Tunik E, Turk D, Inati S, et al. The angular gyrus computes action awareness representations. Cereb Cortex (2007) [Epub ahead of print].
Fine I, Finney EM, Boynton GM, Dobkins KR. Comparing the effects of auditory deprivation and sign language within the auditory and visual cortex. J Cogn Neurosci (2005) 17:1621–37.[CrossRef][Web of Science][Medline]
Finney EM, Clementz BA, Hickok G, Dobkins KR. Visual stimuli activate auditory cortex in deaf subjects: evidence from MEG. Neuroreport (2003) 14:1425–7.[CrossRef][Web of Science][Medline]
Ghazanfar AA, Maier JX, Hoffman KL, Logothetis NK. Multisensory integration of dynamic faces and voices in rhesus monkey auditory cortex. J Neurosci (2005) 25:5004–12.
Ghazanfar AA, Schroeder CE. Is neocortex essentially multisensory? Trends Cogn Sci (2006) 10:278–85.[CrossRef][Web of Science][Medline]
Giraud AL, Price CJ, Graham JM, Truy E, Frackowiak RS. Cross-modal plasticity underpins language recovery after cochlear implantation. Neuron (2001) 30:657–63.[CrossRef][Web of Science][Medline]
Gold BT, Buckner RL. Common prefrontal regions coactivate with dissociable posterior regions during controlled semantic and phonological tasks. Neuron (2002) 35:803–12.[CrossRef][Web of Science][Medline]
Hall DA, Fussell C, Summerfield AQ. Reading fluent speech from talking faces: typical brain networks and individual differences. J Cogn Neurosci (2005) 17:939–53.[CrossRef][Web of Science][Medline]
Hillis AE, Newhart M, Heidler J, Barker PB, Herskovits EH, Degaonkar M. Anatomy of spatial attention: insights from perfusion imaging and hemispatial neglect in acute stroke. J Neurosci (2005) 25:3161–7.
Jacquemot C, Pallier C, LeBihan D, Dehaene S, Dupoux E. Phonological grammar shapes the auditory cortex: a functional magnetic resonance imaging study. J Neurosci (2003) 23:9541–6.
Koechlin E, Ody C, Kouneiher F. The architecture of cognitive control in the human prefrontal cortex. Science (2003) 302:1181–5.
Lafon JC. Le test phonétique et la mesure de l’audition—Eindheren, ed. (1964) Centrex.
Lambertz N, Gizewski ER, de Greiff A, Forsting M. Cross-modal plasticity in deaf subjects dependent on the extent of hearing loss. Brain Res Cogn Brain Res (2005) 25:884–90.[CrossRef][Medline]
Lehmann S, Murray MM. The role of multisensory memories in unisensory object discrimination. Brain Res Cogn Brain Res (2005) 24:326–34.[CrossRef][Medline]
Levanen S, Hamdorf D. Feeling vibrations: enhanced tactile sensitivity in congenitally deaf humans. Neurosci Lett (2001) 301:75–7.[CrossRef][Web of Science][Medline]
Ludman CN, Summerfield AQ, Hall D, Elliott M, Foster J, Hykin JL, et al. Lip-reading ability and patterns of cortical activation studied using fMRI. Br J Audiol (2000) 34:225–30.[Web of Science][Medline]
Lyxell B, Ronnberg J, Samuelsson S. Internal speech functioning and speechreading in deafened and normal hearing adults. Scand Audiol (1994) 23:179–85.[Web of Science][Medline]
MacLeod A, Summerfield Q. Quantifying the contribution of vision to speech perception in noise. Br J Audiol (1987) 21:131–41.[Medline]
MacSweeney M, Campbell R, Woll B, Giampietro V, David AS, McGuire PK, et al. Dissociating linguistic and nonlinguistic gestural communication in the brain. Neuroimage (2004) 22:1605–18.[CrossRef][Web of Science][Medline]
MacSweeney M, Woll B, Campbell R, McGuire PK, David AS, Williams SC, et al. Neural systems underlying British Sign Language and audio-visual English processing in native users. Brain (2002) 125:1583–93.
Meredith MA, Stein BE. Spatial determinants of multisensory integration in cat superior colliculus neurons. J Neurophysiol (1996) 75:1843–57.
Moradi F, Hipp C, Koch C. Activity in the visual cortex is modulated by top-down attention locked to reaction time. J Cogn Neurosci (2007) 19:331–40.[CrossRef][Web of Science][Medline]
Oldfield RC. The assessment and analysis of handedness: the Edinburgh inventory. Neuropsychologia (1971) 9:97–113.[CrossRef][Web of Science][Medline]
Pekkola J, Ojanen V, Autti T, Jaaskelainen IP, Mottonen R, Sams M. Attention to visual speech gestures enhances hemodynamic activity in the left planum temporale. Hum Brain Mapp (2006) 27:471–7.[CrossRef][Web of Science][Medline]
Petitto LA, Zatorre RJ, Gauna K, Nikelski EJ, Dostie D, Evans AC. Speech-like cerebral activity in profoundly deaf people processing signed languages: implications for the neural basis of human language. Proc Natl Acad Sci USA (2000) 97:13961–6.
Puce A, Allison T, Bentin S, Gore JC, McCarthy G. Temporal cortex activation in humans viewing eye and mouth movements. J Neurosci (1998) 18:2188–99.
Reilly KT, Mercier C, Schieber MH, Sirigu A. Persistent hand motor commands in the amputees’ brain. Brain (2006) 129:2211–23.
Rouger J, Lagleyre S, Fraysse B, Deneve S, Deguine O, Barone P. From the cover: evidence that cochlear-implanted deaf patients are better multisensory integrators. Proc Natl Acad Sci USA (2007) 104:7295–300.
Schorr EA, Fox NA, van Wassenhove V, Knudsen EI. Auditory-visual fusion in speech perception in children with cochlear implants. Proc Natl Acad Sci USA (2005) 102:18748–50.
Sheffert SM, Olson E. Audiovisual speech facilitates voice learning. Percept Psychophys (2004) 66:352–62.[Web of Science][Medline]
Scott SK, Johnsrude IS. The neuroanatomical and functional organization of speech perception. Trends Neurosci (2003) 26:100–7.[CrossRef][Web of Science][Medline]
Smirnakis SM, Brewer AA, Schmid MC, Tolias AS, Schuz A, Augath M, et al. Lack of long-term cortical reorganization after macaque retinal lesions. Nature (2005) 435:300–7.[CrossRef][Medline]
Summerfield Q. Use of visual information for phonetic perception. Phonetica (1979) 36:314–31.[Web of Science][Medline]
Summerfield Q. Lipreading and audio-visual speech perception. Philos Trans R Soc Lond B Biol Sci (1992) 335:71–8.
Tyler RS, Lowder MW, Parkinson AJ, Woodworth GG, Gantz BJ. Performance of adult ineraid and nucleus cochlear implant patients after 3.5 years of use. Audiology (1995) 34:135–44.[Web of Science][Medline]
van Dijk JE, van Olphen AF, Langereis MC, Mens LH, Brokx JP, Smoorenburg GF. Predictors of cochlear implant performance. Audiology (1999) 38:109–16.[Web of Science][Medline]
van Wassenhove V, Grant KW, Poeppel D. Visual speech speeds up the neural processing of auditory speech. Proc Natl Acad Sci USA (2005) 102:1181–6.
Vigneau M, Beaucousin V, Herve PY, Duffau H, Crivello F, Houde O, et al. Meta-analyzing left hemisphere language areas: phonology, semantics, and sentence processing. Neuroimage (2006) 30:1414–32.[CrossRef][Web of Science][Medline]
von Kriegstein K, Giraud AL. Implicit multisensory associations influence voice recognition. PLoS Biol (2006) 4:e326.[CrossRef][Medline]
von Kriegstein K, Kleinschmidt A, Giraud AL. Voice recognition and cross-modal responses to familiar speakers’ voices in prosopagnosia. Cereb Cortex (2006) 16:1314–22.
von Kriegstein K, Kleinschmidt A, Sterzer P, Giraud AL. Interaction of face and voice areas during speaker recognition. J Cogn Neurosci (2005) 17:367–76.[CrossRef][Web of Science][Medline]
Womelsdorf T, Anton-Erxleben K, Pieper F, Treue S. Dynamic shifts of visual receptive fields in cortical area MT by spatial attention. Nat Neurosci (2006) 9:1083–4.[CrossRef][Web of Science][Medline]
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  K. Strelnikov, J. Rouger, J.-F. Demonet, S. Lagleyre, B. Fraysse, O. Deguine, and P. Barone Does Brain Activity at Rest Reflect Adaptive Strategies? Evidence from Speech Processing after Cochlear Implantation Cereb Cortex, October 5, 2009; (2009) bhp183v1. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 M.-W. Suh, H.-J. Lee, J. S. Kim, C. K. Chung, and S.-H. Oh Speech experience shapes the speechreading network and subsequent deafness facilitates it Brain, October 1, 2009; 132(10): 2761 - 2771. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 B. L. Allman, L. P. Keniston, and M. A. Meredith Adult deafness induces somatosensory conversion of ferret auditory cortex PNAS, April 7, 2009; 106(14): 5925 - 5930. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 A. J King Visual influences on auditory spatial learning Phil Trans R Soc B, February 12, 2009; 364(1515): 331 - 339. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||