Brain Advance Access published online on February 7, 2008
Brain, doi:10.1093/brain/awn003
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Cross-modal localization in hemianopia: new insights on multisensory integration
1Dipartimento di Psicologia, Università degli Studi di Bologna, Bologna, 2CSRNC, Centro Studi e Ricerche in Neuroscienze Cognitive, Cesena, 3Dipartimento di Psicologia, Università degli Studi di Milano-Bicocca, Milano, Italy and 4Department of Neurobiology and Anatomy, Wake Forest University School of Medicine, Winston-Salem, NC, USA
Correspondence to:
Correspondence to: Elisabetta Ladavas, Department of Psychology, University of Bologna, Viale Berti Pichat, 5 - 40127 Bologna, Italy E-mail: elisabetta.ladavas{at}unibo.it
 |
Summary |
|---|
 Top Summary IntroductionExperiment 1Experiment 2General discussionReferences |
|---|
The superior colliculus (SC) has been implicated in the mediation of residual visual function in hemianopic patients, and has been shown to be capable of using multiple sensory cues to facilitate its localization functions. The aim of the present study was to examine the possibility that the SC could effect covert visual processes, via multisensory integration of auditory and visual stimuli in patients with visual field loss.
To this aim hard-to-localize auditory targets were presented alone (unimodal condition) or with a visual stimulus (cross-modal condition) in either hemifield and at various spatial (0°, 16°, 32°) and temporal (0 ms, 500 ms) disparities. The results showed substantial field-specific differences. As expected, a visual stimulus in the intact hemifield induced a strong visual bias in auditory localization independent of the spatial disparities, and did so even when the two stimuli were temporally offset. In these spatially disparate conditions, the localization accuracy was markedly reduced. In the blind hemifield, however, the visual stimulus affected auditory localization only when it was coincident with that target in both space and time. In this circumstance auditory localization performance was markedly enhanced.
This result strongly suggests that covert visual processes remain active in hemianopia, though they differ from those in the normal hemifield. A likely explanation of these differences is that enhancement and visual bias depend on different neural pathways: with the former dependent on circuits involving the superior colliculus, a structure involved in the integration of cues from multiple senses to facilitate orientation/localization; and the latter dependent on geniculo-striate circuits that facilitate more detailed analyses of the visual scene.
Overall the present results not only enhance our understanding of the impact of covert visual processes in hemianopic patients, but also enhance our knowledge of how different brain regions areas contribute to processing cross-modal information.
Key Words: blindsight; hemianopia; multisensory localization; superior colliculus; visual cortex
Abbreviations: dLGN, dorsal lateral geniculate nucleus; DTI, diffusion tensor imaging; SC, superior colliculus
Received May 22, 2007. Revised December 12, 2007. Accepted January 3, 2008.
|
Introduction |
|---|
|
TopSummaryIntroductionExperiment 1Experiment 2General discussionReferences |
|---|
Homonymous hemianopia is a contralesional visual field defect. It generally results from damage to visual cortex on one side of the brain and it is manifested as a loss of vision in one hemifield, which corresponds retinotopically to the damaged area (Zihl and Kennard, 1996
). Homonymous field disorders differ in gravity, depending on the extent and completeness of the brain damage, and in the most extreme case hemianopic patients exhibit complete and persistent ‘cortical’ blindness. In effect, an apparent scotoma has been produced in their contralesional visual field, wherein they have no conscious vision but may retain neuroendocrine, reflexive and other behavioural responses to visual stimuli (Stoerig and Cowey, 1997).
Residual visual functions without acknowledged awareness after retrogeniculate lesions have been referred to as ‘blindsight’ (Weiskrantz, 1986). Two methodologically distinct classes of visual responses in blindsight have been identified: the implicit and the direct (or explicit). In the implicit condition a visual stimulus in the blind field affects responses to a visual stimulus in the normal field. This type of blindsight is considered to represent a comparatively low level of visual information processing and is of particular interest here. An explicit response involves a direct report about stimuli in the affected region and is believed to represent the highest level of unconscious visual function (Weiskrantz, 1986; Stoerig, 1996; Stoerig and Cowey, 1997).
A new taxonomy for the subtypes of residual abilities described in blindsight was recently proposed by Danckert and Rossetti (2005). This includes: ‘action blindsight’ which refers to the ability to accurately act upon blind field stimuli (e.g. by pointing or making saccadic eye movements towards them); and ‘attentional blindsight’ which refers to some aspects of covert spatial orienting including inhibition of return and implicit task interference effect (e.g. the flanker task with flankers presented to the blind field) (Danckert et al., 1998; Kentridge et al., 1999; Walker et al., 2000) that rely on attentional processes but are not necessarily associated with a specific action or effectors. Finally there is ‘agnosopsia’ (Zeki and Ffytche, 1998), which refers to the ability to correctly describe the characteristics of blind field stimuli despite the complete lack of a conscious percept. This would include residual visual abilities that involve form and wavelength discrimination.
This classification system assumes that blindsight is a complex of capabilities mediated by a number of pathways that target different extrastriate regions of cortex. Thus, ‘agnosopsia’ is presumably mediated by projections derived from interlaminar layers of the dorsal lateral geniculate nucleus (dLGN). Action blindsight and attentional blindsight are mediated by projections from the superior colliculus (SC); the former likely terminates in areas of dorsal extrastriate and posterior parietal cortices, known to be important for the control of visually guided actions (Danckert and Goodale, 2000), while the targets of the latter are less clear (Danckert and Rossetti, 2005).
In a recent study using diffusion tensor imaging (DTI) tractography, Leh and coworkers (Leh et al., 2006) have reconstructed SC projections in hemispherectomized patients, with massive removal or disconnection of an entire cerebral hemisphere including the occipital lobe. Some of these patients exhibited attentional blindsight, others did not. In those patients with ‘attentional blindsight’, ipsilateral and contralateral connections from the SC to widespread areas of remaining visual and visuomotor cortex were retained; no such connections were identified in those hemispherectomized patients without ‘attentional blindsight’. Control subjects demonstrated mainly ipsilateral connections to cortical areas. Thus, attentional blindsight is likely to be mediated by spared SC-cortical projections, some of which may have reorganized. It is also possible that such reorganization can take place also after circumscribed lesion to primary visual areas.
The aim of the present study was to examine the possibility that the SC could effect covert visual processes, via multisensory integration of auditory and visual stimuli in patients with visual field loss. The SC has been implicated in the mediation of residual visual function in such patients (Leh et al., 2006), and has been shown to be capable of using multiple sensory cues to facilitate its localization functions (Stein and Meredith, 1993; Stein, 1998).
One of the most striking effects in multisensory localization behaviour is the ability of a visual cue to strongly bias the perceived location of a weak auditory stimulus: the sound is translocated toward the visual cue (Bertelson and Radeau, 1981; Slutsky and Recanzone, 2001; Hairston et al., 2003b; Wallace et al., 2004). This visual dominance is generally the case, presumably because of its superior localization accuracy (Welch and Warren, 1980). This effect is though to be mediated by a projection from visual to the auditory cortex (Bonath et al., 2007). Although there is no reason to suppose that this effect would be altered in the normal visual hemifield of cortically blind patients, it cannot be retained in the blind field due to the presence of an occipital lesion. However, a damage of the occipital cortex does not eliminate responses at the level of SC. For example, visual orientation in hemianopic cats can be restored by modulating SC activity (Ciaramitaro, 1997). Because of the crucial role of the SC in mediating multisensory integration and spatial orienting behaviours in animals (Stein and Meredith, 1993), multisensory enhancement of auditory localization in the blind field of humans might be possible. If it did indeed occur, it would be expected to follow the same principles that govern SC-mediated multisensory behaviours in animals (Stein and Meredith, 1993). In this context it is important to remember that the enhanced responses typical of SC multisensory neurons are particularly strong when weakly effective stimuli are combined, whereas the proportionate magnitude of multisensory enhancement is much reduced or absent for the pairing of strongly effective stimuli (the so-called inverse effectiveness rule) (Stein and Meredith, 1993). In this respect, hemianopic patients are particularly well suited for exploring the nature of such an ‘inverse effectiveness rule’ because the visual stimuli in the intact and blind field have a substantially different efficacy.
There are two additional principles that govern multisensory integration in the SC that are important here: these are the so-called spatial and temporal principles, whereby neuronal activity is modulated by the spatial and temporal arrangement of the bimodal stimuli. Only cross-modal stimuli presented in close spatial and temporal proximity induce a response enhancement, while spatially and temporally disparate stimuli produce depression or no change in neuronal activity. Based on these rules of multisensory integration one would predict that auditory localization performance in the blind hemifield would be enhanced only when the cross-modal stimuli are spatially and temporally coincident (Meredith and Stein, 1986a, b; Kadunce et al., 2001).
The prediction in the present experiments is that visual stimuli in the intact visual field will bias the auditory localization so that sounds will be mislocated toward their apparent (and disparate) visual source. This is in line with the well-documented visual bias effect (Howard and Templeton, 1966; Welch and Warren, 1980; Hairston et al., 2003b). This visual bias is expected regardless of whether the cross-modal stimuli are simultaneous or temporally disparate (SOA 500), although with a different magnitude (Hairston et al., 2003a). In contrast, such an effect is not expected, in the blind field, where the occipital cortex damage has disrupted its underlying neural circuitry. Here, however, enhancement of auditory localization is expected via SC neurons whenever the cross-modal stimuli are in spatial and temporal concordance (Stein and Meredith, 1993).
|
Experiment 1 |
|---|
|
TopSummaryIntroductionExperiment 1Experiment 2General discussionReferences |
|---|
Materials and methods
Patients
Twelve brain-damaged patients were selected based on their performance on a visual perimetry test (Fig. 1). All had chronic visual field defects. They gave informed consent to participate according to the Declaration of Helsinki (BMJ 1991; 302: 1194) and the Ethical Committee of the Department of Psychology, University of Bologna. Details concerning sex, age, length of illness, lesion sites and the presence of visual field defect are reported in Table 1 and in Fig. 1.
|
|
All patients had suffered lesions, as confirmed by CT/MRI scanning (Table 2), and all but one patient (i.e. P5) had deficits confined to one hemifield (Fig. 1).
|
All patients were right-handed, alert, cooperative and well oriented in space and time, as documented by normal Milan Overall Dementia Assessment (MODA) (Brazzelli et al., 1994a
, b) scores. Before the experiment, a neuropsychological examination was carried out to assess the presence of visual and auditory deficits (Bolognini et al., 2005b). The presence of a visual field defect was also evaluated using the apparatus itself (see ‘Materials’ section and Fig. 2). In that procedure, a visual target was presented for 100 ms in each of the eight spatial positions (see later). Ninety-six trials were presented: 10 trials in each visual position and 16 trials in which no visual stimulus was presented (i.e. ‘catch trials’). The total number of trials was equally distributed across two blocks. Patients were instructed to press a response button to indicate the presence of a visual target. All patients were able to detect the presence of nearly all ipsilesional visual stimuli regardless of their specific location (mean correct detection = 90%), but were severely impaired in detecting contralesional visual stimuli (mean percentage = 0.3%, mean false alarms = 0%, see Table 3).
|
|
Patients showed normal hearing thresholds, as measured by audiometry in each ear, with no sign of interaural asymmetry. Auditory detection performance was tested using a paradigm similar to that used for visual detection, and all patients detected the auditory stimulus with 100% of accuracy in both hemifields regardless of its specific spatial position.
Materials
The apparatus consisted of a plastic semicircular perimetry device (height 40 cm, length 200 cm) that was fixed to the surface of a table (Fig. 2). Eight piezoelectric loudspeakers (0.4 W, 8
) were located horizontally at ear level, at eccentricities of 24°, 40°, 56° and 72° to the left and right of the central fixation point (0°). They are referred to as A1 to A8 moving from left to right. A black fabric curtain hid the speakers from view.
The auditory targets consisted of a 100 ms burst of pure tones (4000 Hz) with an intensity range of 76.5–58.5 dB. Before each testing session, the auditory target intensities were varied in order to obtain subjectively hard-to-localize stimuli for each subject (see later). Each auditory target was presented against a constant background noise (56.5 dB) generated by two white-noise generators located behind the experimental apparatus at 32° to the left and right of the fixation point.
Prior to the experiment, auditory stimuli were calibrated for each patient so as to be difficult to localize. Difficulty was assessed using each patient's mean localization error (i.e. the difference between actual and reported location). The criterion for stimulus selection was a sound with a localization error of >8° on at least 50% of the trials. If a lower localization error was obtained, uncertainty was added by decreasing sound intensity (range = 76.5–58.5 dB).
The visual stimuli were generated by single green light-emitting diodes (LEDs, 90 cd/m2) poking out of the black fabric, and were presented directly in front of each speaker (the visual stimuli are referred as V1 to V8, moving left to right). Although never stimulated, other LEDs were located at 32°, 48° and 64° to either side of the central fixation to increase uncertainty in the localization task (Fig. 2).
All visual and auditory stimuli were 100 ms pulses. Their timing, and response acquisition were controlled by an ACER 711TE laptop computer, using a custom program (XGen – Experimental Software, http://www.psychology.nottingham.ac.uk/staff/cr1/) and a custom hardware interface.
Experimental procedure
The experiment was conducted in a dimly lit, sound-attenuated room. The patient sat in front of the apparatus, at a distance of 70 cm. He/she faced directly ahead, with the body aligned with the centre of the apparatus. Before each trial, the patient fixated a 1° white triangle located at 0°. The experimenter stood behind the apparatus, facing the patient and assessed fixation on each trial. Any trials in which fixation was not maintained or was questionable were eliminated from consideration. The experiment was carried out under binocular vision.
Three testing conditions were used:
- Unimodal auditory condition (A): the auditory stimulus was presented alone.
- Unimodal visual catch-trial condition: the visual stimulus was presented alone.
- Cross-modal condition: the auditory stimulus was presented at each location together with a temporally coincident visual stimulus. The visual stimulus was either spatially coincident (SP, same position) or spatially disparate with the auditory target (DP, different positions). Spatial disparities were 16° (DP-16°) or 32° (DP-32°) (nasal or temporal).
The following trials were presented: 120 unimodal auditory and 120 unimodal visual trials (15 for each of the 8 positions); 120 spatially coincident cross-modal trials (15 for each of the 8 positions); and 240 spatially disparate cross-modal trials (15 for each of the 16 cross-modal spatially disparate conditions). The total number of trials was 600, and these were equally distributed in 15 experimental blocks (40 trials each) over two consecutive days.
Because cortical lesions can disrupt egocentric coordinates and bias manual responses (Farne et al., 1998; Ladavas and Pavani, 1998; Pavani et al., 2003), a pointing task was avoided here. Rather, patients were required to respond verbally. The apparatus was marked in clearly visible 1° steps from left to right and numbered sequentially from 1 to 72. Patients were instructed to report the number corresponding to the location of the sound, and to ignore any accompanying visual stimulus.
Results
Performance was evaluated for responses to auditory stimuli at only two spatial positions: A2/A3 (56° and 40° in the left visual field, LVF), and A6/A7 (40° and 56° in the right visual field, RVF). Auditory stimuli were presented at more peripheral locations (i.e. A1 and A8) to increase the patients’ uncertainty as to the location of the auditory stimulus, but these were not analysed. This was because more peripheral localization judgements were not possible in these circumstances and the inclusion of these locations in the analysis would have produced a nasal response bias in the data set. For similar reasons responses to A4 and A5 were not analysed. Because no false alarms were noted on catch trials, they, too, were eliminated from consideration.
Auditory localization accuracy
Localization error was calculated in both unimodal auditory and cross-modal conditions as the absolute difference, expressed in degrees, between the verbal localization response and the actual target location. To amplify the power of the sample and reduce the number of comparisons, data were collapsed across positions in each hemifield (i.e. A2–A3 for LVF and A6–A7 for RVF). This procedure is justified by the fact that no differences between spatial positions were found in a preliminary analysis. Data were then analysed using a repeated-measures analysis of variance (two-way ANOVA) with Condition (i.e. unimodal auditory versus cross-modal) and Hemifield (i.e. hemianopic versus intact field) as main factors. Cross-modal conditions included those in which a visual stimulus was either spatially coincident or 16° or 32° disparate to the auditory stimulus. Pairwise comparisons were conducted using the Duncan test.
A significant main effect Condition [F(3,33) = 30.76, P < 0.000001] was clearly evident. More interesting, we found a significant interaction between Hemifield and Condition [F(3,33) = 19.35, P < 0.000001]. In the hemianopic field, a spatially coincident visual stimulus significantly reduced the localization error established in the unimodal condition (10° versus 12°, P < 0.03, respectively). However, this reduction was evident only in the spatially coincident condition; when compared to the localization error in the unimodal condition (A = 12°), we found no difference in errors in cross-modal spatially disparate conditions (DP-16° = 12°; DP-32° = 12°).
In the intact hemifield, the reduction in the localization error was of similar magnitude as found in the hemianopic field (t-test one-tail, P < 0.3), when the bimodal stimuli were presented in the same spatial position (SP = 8° versus A = 11°, P < 0.002). In contrast, comparing to the unimodal auditory condition (A = 11°), visual stimuli presented in disparate positions increased localization errors to 13° in the 16° disparity condition (P < 0.05) and to 19° in the 32° disparity condition (P < 0.00003) (Fig. 3A).
|
It was therefore apparent that only a spatially coincident visual stimulus affected auditory localization in the hemianopic field, and that it did so by enhancing performance. The degradation of auditory localization induced by a disparate visual stimulus evident in the intact field, was not evident in the blind field. This loss of visual biasing is presumably a consequence of the cortical lesion. In order to verify this result and measure the magnitude of the visual biasing, a second analysis was carried out.
Visual bias of auditory location
Visual bias was calculated by subtracting the actual location of the sound from the average location reported, dividing the result by the actual visual-auditory disparity and multiplying by 100 (Hairston et al., 2003b; Wallace et al., 2004). The resulting percentage score represents the degree of ‘visual bias’ of sound location. Note that visual bias cannot be computed in the absence of disparity. Hence, a score of 100% represents complete bias, wherein the subject localizes the sound at the visual stimulus site, whereas positive scores <100% indicate position judgements between the visual and auditory stimuli, and negative scores reflect position judgements on the opposite side of the auditory stimulus.
Data were collapsed across positions in each hemifield (i.e. A2–A3 for LVF and A6–A7 for RVF) and the data were then analysed with a two-way ANOVA. Hemifield (hemianopic versus intact) and spatial disparity (16° versus 32°) were the main factors. A significant main effect was obtained only for Hemifield [F(1,11) = 36.87, P < 0.00009], showing that regardless of the magnitude of spatial disparity, only in the intact hemifield did the visual stimulus bias sound localization (visual bias in the intact visual field = 41%; visual bias in hemianopic field = 1%; see Fig. 3B).
Discussion
The result of the present study showed that the localization of an auditory target in the blind field can be improved by presenting a visual stimulus in the same spatial position. After demonstrating that, in the hemianopic field, spatial coincidence of bimodal stimuli is necessary for enhancement in localization accuracy, Experiment 2 tested whether this effect also depended on temporal coincidence. To this end, we employed the same methodological design as in Experiment 1, but in the cross-modal conditions the visual stimulus always preceded the auditory one by 500 ms. According to temporal rule, the enhancement of auditory localization in the hemianopic field should disappear, given that the SOA employed (500 ms) goes beyond the window of integration of multisensory neurons in the SC (Stein and Meredith, 1993). Instead, the visual bias in the intact field, although reduced, should still be evident; in fact, a substantial visual bias was still found with larger temporal separations (800 ms) (Wallace et al., 2004). In Experiment 2 we also used a more conservative task (two-alternative forced-choice task, 2AFC) to ascertain the lack of overt processing of the visual stimuli.
|
Experiment 2 |
|---|
|
TopSummaryIntroductionExperiment 1Experiment 2General discussionReferences |
|---|
Materials and methods
Patients
Seven of the 12 patients that took part in Experiment 1 participated in Experiment 2 (P6, P7, P8, P9, P10, P11, P12).
In these patients the presence of residual processing of visual stimuli in the hemianopic field was also evaluated by a two-alternative forced-choice task, using the experimental apparatus. Stimuli were presented in four blocks of 60 trials, one block for each of the four different spatial positions in the hemianopic field. In each block, 50% of trials were target-present conditions, whereas the others were target-absent conditions. Patients were asked to press one of the two response buttons to indicate the presence or absence of the visual target. All patients responded randomly regardless of the presence or absence of the visual stimuli and of their specific location (Table 4).
|
Materials
The experimental apparatus was identical to the one used in Experiment 1 (Fig. 2).
Experimental procedure
The experimental procedure was the same as in Experiment 1, except that in the cross-modal conditions the visual stimulus always preceded the auditory one by 500 ms (temporally disparate cross-modal condition).
Three testing conditions were used:
- Unimodal auditory condition (A): the auditory stimulus was presented alone.
- Unimodal visual catch-trial condition: the visual stimulus was presented alone.
- Cross-modal condition: the auditory stimulus was presented at each location 500 ms after a visual stimulus. The visual stimulus was either spatially coincident (SP) or spatially disparate with the auditory target. Spatial disparities were 16° (DP-16°) or 32° (DP-32°) (nasal or temporal).
Results
Auditory localization accuracy
Localization error was calculated in the same way as in Experiment 1. Data related to temporally coincident cross-modal conditions were derived from Experiment 1.
In order to compare the effect of both the spatial and the temporal determinant in auditory localization performance, data were analysed with a repeated-measures analysis of variance (three-way ANOVA) with SOA (i.e. 0 versus 500 ms), Condition (i.e. unimodal versus cross-modal) and Hemifield (i.e. hemianopic versus intact hemifield) as main factors.
As in Experiment 1, a significant effect of the main factor Condition [F(3,18) = 19.67, P < 0.000008] was evident. More interestingly a significant interaction between SOA, Condition and Hemifield was found [F(3,18) = 5.42, P < 0.008].
In the hemianopic field, when the visual stimulus was presented 500 ms before the auditory target (SOA = 500 ms), there was no significant difference in the auditory localization accuracy between the unimodal condition (12°) and the cross-modal conditions (SP = 12°; DP-16° = 12°; DP-32° = 13°).
In contrast, when the bimodal stimuli were presented with both temporal (SOA = 0 ms) and spatial coincidence, the localization error was significantly reduced compared to the unimodal condition (10° versus 13°, P < 0.02, respectively). As in Experiment 1, no difference was obtain between unimodal (13°) and cross-modal spatially disparate conditions (DP-16° = 13°; DP-32° = 12°) (Fig. 4A).
|
With respect to the intact field, the introduction of a temporal asynchrony between the two stimuli reduced the strength of the visual bias. Compared to the unimodal auditory condition (A = 9°) a visual stimulus delivered 500 ms before the auditory target did only marginally bias the localization performance in the cross-modal conditions (SP = 8°, P = 0.08; DP-16° = 11°, P = 0.06; DP-32° = 14°, P < 0.0002). In contrast, the simultaneous presentation of a visual stimulus (SOA 0) resulted to strongly bias auditory localization performance; comparing to unimodal condition (9°) the accuracy in the spatially coincident condition increased (5°, P < 0.0006), and decreased in each spatially disparate conditions (DP-16° = 12°, P < 0.02; DP-32° = 18°, P < 0.00002; see Fig. 4C).
Visual bias of auditory location
The visual bias was computed in the same way as described in the Experiment 1. Data were analysed with a repeated-measures analysis of variance (three-way ANOVA) with SOA (i.e. 0 versus 500 ms), Hemifield (hemianopic versus intact) and Spatial Disparity (16° versus 32°) as main factors. A significant main effect of Hemifield [F(1,6) = 8.28, P < 0.03] and a significant interaction between SOA and Hemifield [F(1,6) = 9.83, P < 0.03] were found.
In the hemianopic field, the introduction of a temporal delay of 500 ms did not have any effect on the amount of visual bias when compared to the condition of simultaneity (SOA 500 = 7.3% versus SOA 0 = 2.5%; P = 0.38; see Fig. 4B).
On the contrary, the magnitude of the visual bias in the intact field was significantly greater when the two stimuli were temporally coincident compared to the temporally disparate condition (SOA 0 = 45% versus SOA 500 = 27%; P < 0.02; see Fig. 4D).
|
General discussion |
|---|
|
TopSummaryIntroductionExperiment 1Experiment 2General discussionReferences |
|---|
The aim of the present study was to examine the impact of covert visual processes on multisensory integration. Comparing the effect of audio–visual stimulation in the two hemifields of hemianopic patients offered the opportunity to evaluate whether conscious (within the ipsilesional hemifield) and unconscious vision (within the controlesional hemifield) differentially influence sensory integration.
Here we showed that visual information in the blind hemifield of hemianopic patients can significantly improve their auditory localization performance, but only when the two stimuli are both spatially and temporally coincident. Surprisingly, this multisensory benefit was as great in the blind field as in the intact hemifield. Nevertheless, patients remained unaware of both the presence of the visual stimulus in their blind field, and of its effects on their auditory responses. These observations are consistent with speculation that the SC, spared in our patients, participates in some aspects of blindsight (Rafal et al., 1990; Stoerig and Cowey, 1997; Azzopardi and Cowey, 1998; Schoenfeld et al., 2002; Ro et al., 2004; Leh et al., 2006). The novel finding here is that covert visual processes in hemianopic patients are not restricted to within-modal visual activity; blind field visual stimuli can enhance the perception of stimuli from another sensory modality. The result of this multisensory integration is an improvement in stimulus localization, which is likely mediated, at least in part, by multisensory neurons in the SC (Stein and Meredith, 1993).
Despite the retention of some visual-auditory integration in the hemianopic field, the visual biasing of auditory localization was not retained. Gone was the normal visual capture that occurs when these cross-modal stimuli are spatially disparate. Though evident in the intact field, there was no evidence that the auditory stimulus was perceptually translocated by a visual stimulus in the hemianopic field. Similar results were found on normal subjects by using the same experimental procedure, with the exception that a visual stimulus was presented either at or above threshold. The results reveal that stimulus salience is a critical factor in determining the effect of a neutral visual cue on auditory localization. Visual capture, and hence perceptual translocation of the auditory stimulus, occurred when the visual stimulus was supra-threshold, regardless of its location. However, this was not the case when the visual stimulus was at threshold. In those trials, the influence of the visual cue was apparent only when the two cues were spatially coincident and resulted in an enhancement of stimulus localization (Bolognini et al., 2008).
The observations on hemianopic patients suggest that the neural circuitry underlying the visual influences on auditory localization differs depending on whether the stimuli are likely to be derived from the same event (i.e. are spatially coincident) or from different events. Apparently, the damaged visual cortices were critical for the latter, but not for the former. This proposal is indeed supported by a recent finding showing that the ventriloquism illusion is directly related to the visual influences on the auditory cortex responses to sound (Bonath et al., 2007). The absence of a visual bias in the hemianopic field when the two stimuli are spatially separated supports the key role of visual cortex for such an effect; when the visual cortex has been damaged no visual bias is observed.
In contrast, the lesion of the visual cortex does not prevent the integration of the two auditory and visual stimuli, when they are simultaneously presented in the same spatial position. Instead, a temporal interval of 500 ms abolished the integration effect. Based on an analogous visual-auditory localization paradigm, neurophysiological findings in animals with reversible or permanent lesions of association cortex support our results. The ability of SC neurons to integrate cross-modal inputs in cats is known to involve descending inputs from specific regions of association cortex (the anterior ectosylvian sulcus, AES; and rostral lateral suprasylvian sulcus, rLS), and not from primary or secondary cortices (Wallace and Stein, 1994; Stein, 2005). Presumably, whatever the homologous regions are in humans, they would be distant from the hemianopic-inducing lesions in the patients studied here. Jiang and coworkers (Jiang et al., 2001, 2002; Jiang and Stein, 2003) suggest that while the enhancement of spatially coincident visual-auditory stimuli is eliminated in cats by AES and rLS lesions, the depression normally induced by disparate visual-auditory stimuli is only degraded because these competitive or depressive SC functions involve contribution from other brain areas.
The present observations are also consistent with the parallels among perspectives guiding our understanding of visual dominance and multisensory integration. For example, it has been proposed (Heron et al., 2004) that when the nervous system deems visual information most reliable, as is generally the case in normal conditions, vision dominates auditory localization judgements. As noted earlier, the impaired perceptual nature of visual stimuli in the hemianopic visual field eliminates such perceptual dominance, but not the spatial, the temporal and the inverse effectiveness rules of multisensory integration (Stein and Meredith, 1993). These rules, evident at both the physiological (Meredith and Stein, 1986a, b; Wallace et al., 1996; Kadunce et al., 2001) and behavioural levels (Stein et al., 1988; Frens et al., 1995; Wilkinson et al., 1996; Harrington and Peck, 1998; Hughes et al., 1998; Schroger and Widmann, 1998; Frassinetti et al., 2002a, b; Arndt and Colonius, 2003; Bolognini et al., 2005a, c; Frassinetti et al., 2005) in normal subjects, were also operative in the hemianopic field of brain-damaged patients.
Finally, the results in the intact visual field are in line not only with the phenomenon of visual capture, but also with the related phenomenon known as the ventriloquism effect. In this effect sounds are mislocated toward their apparent visual source in situations in which the cues are spatially disparate (Howard and Templeton, 1966; Thurlow and Jack, 1973; Bertelson and Radeau, 1981; Spence and Driver, 2000; Slutsky and Recanzone, 2001; Hairston et al., 2003a; Lewald and Guski, 2003; Vroomen and De Gelder, 2004; Wallace et al., 2004). Indeed, when the plausibility that a sound originates from a visual object is high, based on the temporal correlation of the stimuli (Jack and Thurlow, 1973) and on cognitive factors (Warren et al., 1981), visual localization can influence auditory localization even when the stimuli are separated by angular distances of as much as 30° (Jackson, 1953; Thurlow and Rosenthal, 1976), findings also in line with the idea that sensory uncertainty determines the perceptual weight allocated to a given cue during multisensory integration (Ernst et al., 2000; Ernst and Banks, 2002; van Beers et al., 2002; Battaglia et al., 2003; Hairston et al., 2003a, b; Alais and Burr, 2004; Heron et al., 2004; Wallace et al., 2004).
In conclusion, the new finding of present study is that an ‘unseen’ visual stimulus presented simultaneously at the same location as the auditory stimulus improves the accuracy of auditory localization. Interestingly, the spatially and temporally coincident visual stimulus affects auditory information processing even when subjects were not able to detect its presence. This finding is presumably related to the inverse effectiveness rule of multisensory integration occurring at a level of SC, by which weakly effective unisensory stimuli have proportionately greater effects than do stronger stimuli (Stein and Meredith, 1993; Frassinetti et al., 2002a, b; Bolognini et al., 2005a, b, c; Frassinetti et al., 2005; Perrault et al., 2005; Stanford et al., 2005). Moreover, our results lend support to the spatial and temporal rules, which suggest that the effective integration occurs only when the visual and auditory stimuli are spatially and temporally coincident. A likely interpretation of our results is that the enhancement of the auditory localization in the blind field reflects the activation of the extrageniculate pathway, leading directly from the retina to the SC, a structure that plays an important role in orientation, localization behaviours and multisensory integration (Stein and Meredith, 1993).
|
Acknowledgements |
|---|
We wish to thank Giuseppe di Pellegrino, Francesca Frassinetti for reading CT and MRI scans and Nancy Stein and Benjamin Rowland for helpful comments on an earlier version of the manuscript. We are also grateful to all the patients for their cooperation. This work was supported by grants from MURST to EL.
|
References |
|---|
|
TopSummaryIntroductionExperiment 1Experiment 2General discussionReferences |
|---|
Alais D, Burr D. The ventriloquist effect results from near-optimal bimodal integration. Curr Biol (2004) 14:257–62.[CrossRef][Web of Science][Medline]
Arndt PA, Colonius H. Two stages in crossmodal saccadic integration: evidence from a visual-auditory focused attention task. Exp Brain Res (2003) 150:417–26.[Web of Science][Medline]
Azzopardi P, Cowey A. Blindsight and visual awareness. Conscious Cogn (1998) 7:292–311.[CrossRef][Web of Science][Medline]
Battaglia PW, Jacobs RA, Aslin RN. Bayesian integration of visual and auditory signals for spatial localization. J Opt Soc Am A Opt Image Sci Vis (2003) 20:1391–7.[Web of Science][Medline]
Bertelson P, Radeau M. Cross-modal bias and perceptual fusion with auditory-visual spatial discordance. Percept Psychophys (1981) 29:578–84.[Web of Science][Medline]
Bolognini N, Frassinetti F, Serino A, Ladavas E. "Acoustical vision" of below threshold stimuli: interaction among spatially converging audiovisual inputs. Exp Brain Res (2005a) 160:273–82.[CrossRef][Web of Science][Medline]
Bolognini N, Rasi F, Coccia M, Ladavas E. Visual search improvement in hemianopic patients after audio-visual stimulation. Brain (2005b) 128:2830–42.
Bolognini N, Rasi F, Ladavas E. Visual localization of sounds. Neuropsychologia (2005c) 43:1655–61.[CrossRef][Web of Science][Medline]
Bolognini N, Leo F, Passamonti C, Stein BE, Làdavas E. Multisensory mediated auditory localisation. Perception (2008 (in press)).
Bonath B, Noesselt T, Martinez A, Mishra J, Schwiecker K, Heinze HJ, Hillyard SA. Neural basis of the ventriloquist illusion. Curr Biol (2007) 17:1697–703.[CrossRef][Medline]
Brazzelli M, Capitani E, Della Sala S, Spinnler H, Zuffi M. A neuropsychological instrument adding to the description of patients with suspected cortical dementia: the Milan overall dementia assessment. J Neurol Neurosurg Psychiatry (1994a) 57:1510–7.
Brazzelli M, Capitani E, Della Sala S, Spinnler H, Zuffi M. Milan Overall Dementia Assessment MODA. Vol. (1994b) Organizzazioni Speciali, Firenze.
Ciaramitaro VM, Todd WE, Rosenquist AC. Disinhinibition of the superior colliculus restores orienting to visual stimuli in the hemianopic field of the cat. J Comp Neurol (1997) 387:568–87.[CrossRef][Web of Science][Medline]
Danckert J, Maruff P, Kinsella G, de Graaff S, Currie J. Investigating form and colour perception in blindsight using an interference task. Neuroreport (1998) 9:2919–25.[Medline]
Danckert J, Goodale MA. A conscious route to unconscious vision. Curr Biol (2000) 10:R64–7.[CrossRef][Web of Science][Medline]
Danckert J, Rossetti Y. Blindsight in action: what can the different sub-types of blindsight tell us about the control of visually guided actions? Neurosci Biobehav Rev (2005) 29:1035–46.[CrossRef][Web of Science][Medline]
Ernst MO, Banks MS, Bulthoff HH. Touch can change visual slant perception. Nat Neurosci (2000) 3:69–73.[CrossRef][Web of Science][Medline]
Ernst MO, Banks MS. Humans integrate visual and haptic information in a statistically optimal fashion. Nature (2002) 415:429–33.[CrossRef][Medline]
Farne A, Ponti F, Ladavas E. In search of biased egocentric reference frames in neglect. Neuropsychologia (1998) 36:611–23.[CrossRef][Medline]
Frassinetti F, Bolognini N, Ladavas E. Enhancement of visual perception by crossmodal visuo-auditory interaction. Exp Brain Res (2002a) 147:332–43.[CrossRef][Web of Science][Medline]
Frassinetti F, Pavani F, Ladavas E. Acoustical vision of neglected stimuli: interaction among spatially converging audiovisual inputs in neglect patients. J Cogn Neurosci (2002b) 14:62–9.[CrossRef][Web of Science][Medline]
Frassinetti F, Bolognini N, Bottari D, Bonora A, Ladavas E. Audiovisual integration in patients with visual deficit. J Cogn Neurosci (2005) 17:1442–52.[CrossRef][Web of Science][Medline]
Frens MA, Van Opstal AJ, Van der Willigen RF. Spatial and temporal factors determine auditory-visual interactions in human saccadic eye movements. Percept Psychophys (1995) 57:802–16.[Web of Science][Medline]
Hairston WD, Laurienti PJ, Mishra G, Burdette JH, Wallace MT. Multisensory enhancement of localization under conditions of induced myopia. Exp Brain Res (2003a) 152:404–8.[CrossRef][Web of Science][Medline]
Hairston WD, Wallace MT, Vaughan JW, Stein BE, Norris JL, Schirillo JA. Visual localization ability influences cross-modal bias. J Cogn Neurosci (2003b) 15:20–9.[CrossRef][Web of Science][Medline]
Harrington LK, Peck CK. Spatial disparity affects visual-auditory interactions in human sensorimotor processing. Exp Brain Res (1998) 122:247–52.[CrossRef][Web of Science][Medline]
Heron J, Whitaker D, McGraw PV. Sensory uncertainty governs the extent of audio-visual interaction. Vision Res (2004) 44:2875–84.[CrossRef][Web of Science][Medline]
Howard IP, Templeton WB. Human spatial orientation. Vol. (1966) London: Wiley.
Hughes HC, Nelson MD, Aronchick DM. Spatial characteristics of visual-auditory summation in human saccades. Vision Res (1998) 38:3955–63.[CrossRef][Web of Science][Medline]
Jack CE, Thurlow WR. Effects of degree of visual association and angle of displacement on the "ventriloquism" effect. Percept Mot Skills (1973) 37:967–79.[Web of Science][Medline]
Jackson C. Visual factors in auditory localization. Q J Exp Psychol (1953) 5:52–65.[Medline]
Jiang W, Wallace MT, Jiang H, Vaughan JW, Stein BE. Two cortical areas mediate multisensory integration in superior colliculus neurons. J Neurophysiol (2001) 85:506–22.
Jiang W, Jiang H, Stein BE. Two corticotectal areas facilitate multisensory orientation behavior. J Cogn Neurosci (2002) 14:1240–55.[CrossRef][Web of Science][Medline]
Jiang W, Stein BE. Cortex controls multisensory depression in superior colliculus. J Neurophysiol (2003) 90:2123–35.
Kadunce DC, Vaughan JW, Wallace MT, Stein BE. The influence of visual and auditory receptive field organization on multisensory integration in the superior colliculus. Exp Brain Res (2001) 139:303–10.[CrossRef][Web of Science][Medline]
Kentridge RW, Heywood CA, Weiskrantz L. Attention without awareness in blindsight. Proc Biol Sci (1999) 266:1805–11.
Ladavas E, Pavani F. Neuropsychological evidence of the functional integration of visual, auditory and proprioceptive spatial maps. Neuroreport (1998) 9:1195–200.[Web of Science][Medline]
Leh SE, Johansen-Berg H, Ptito A. Unconscious vision: new insights into the neuronal correlate of blindsight using diffusion tractography. Brain (2006) 129:1822–32.
Lewald J, Guski R. Cross-modal perceptual integration of spatially and temporally disparate auditory and visual stimuli. Brain Res Cogn Brain Res (2003) 16:468–78.[CrossRef][Medline]
Meredith MA, Stein BE. Spatial factors determine the activity of multisensory neurons in cat superior colliculus. Brain Res (1986a) 365:350–4.[CrossRef][Web of Science][Medline]
Meredith MA, Stein BE. Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. J Neurophysiol (1986b) 56:640–62.
Pavani F, Farne A, Ladavas E. Task-dependent visual coding of sound position in visuospatial neglect patients. Neuroreport (2003) 14:99–103.[CrossRef][Medline]
Perrault TJ Jr, Vaughan JW, Stein BE, Wallace MT. Superior colliculus neurons use distinct operational modes in the integration of multisensory stimuli. J Neurophysiol (2005) 93:2575–86.
Rafal R, Smith J, Krantz J, Cohen A, Brennan C. Extrageniculate vision in hemianopic humans: saccade inhibition by signals in the blind field. Science (1990) 250:118–21.
Ro T, Shelton D, Lee OL, Chang E. Extrageniculate mediation of unconscious vision in transcranial magnetic stimulation-induced blindsight. Proc Natl Acad Sci U S A (2004) 101:9933–5.
Schoenfeld MA, Noesselt T, Poggel D, Tempelmann C, Hopf JM, Woldorff MG, Heinze HJ, Hillyard SA. Analysis of pathways mediating preserved vision after striate cortex lesions. Ann Neurol (2002) 52:814–24.[CrossRef][Web of Science][Medline]
Schroger E, Widmann A. Speeded responses to audiovisual signal changes result from bimodal integration. Psychophysiology (1998) 35:755–9.[CrossRef][Web of Science][Medline]
Slutsky DA, Recanzone GH. Temporal and spatial dependency of the ventriloquism effect. Neuroreport (2001) 12:7–10.[CrossRef][Web of Science][Medline]
Spence C, Driver J. Attracting attention to the illusory location of a sound: reflexive crossmodal orienting and ventriloquism. Neuroreport (2000) 11:2057–61.[Web of Science][Medline]
Stanford TR, Quessy S, Stein BE. Evaluating the operations underlying multisensory integration in the cat superior colliculus. J Neurosci (2005) 25:6499–508.
Stein BE, Huneycutt WS, Meredith MA. Neurons and behavior: the same rules of multisensory integration apply. Brain Res (1988) 448:355–8.[CrossRef][Web of Science][Medline]
Stein BE, Meredith MA. The merging of the senses (1993) Cambridge, Mass; London: MIT Press.
Stein BE. Neural mechanisms for synthesizing sensory information and producing adaptive behaviors. Exp Brain Res (1998) 123:124–35.[CrossRef][Web of Science][Medline]
Stein BE. The development of a dialogue between cortex and midbrain to integrate multisensory information. Exp Brain Res (2005) 166:305–15.[CrossRef][Web of Science][Medline]
Stoerig P. Varieties of vision: from blind responses to conscious recognition. Trends Neurosci (1996) 19:401–6.[CrossRef][Medline]
Stoerig P, Cowey A. Blindsight in man and monkey. Brain (1997) 120(Pt 3):535–59.
Thurlow WR, Jack CE. Certain determinants of the "ventriloquism effect". Percept Mot Skills (1973) 36:1171–84.[Medline]
Thurlow WR, Rosenthal TM. Further study of existence regions for the "ventriloquism effect". J Am Audiol Soc (1976) 1:280–6.[Medline]
van Beers RJ, Wolpert DM, Haggard P. When feeling is more important than seeing in sensorimotor adaptation. Curr Biol (2002) 12:834–7.[CrossRef][Web of Science][Medline]
Vroomen J, De Gelder B. Perceptual effects of cross-modal stimulation: ventriloquism and the freezing phenomenon. In: The handbook of multisensory processes—Calvert G, Spence C, Stein BE, eds. (2004) Cambridge: MIT Press. 141–9.
Walker R, Mannan S, Maurer D, Pambakian AL, Kennard C. The oculomotor distractor effect in normal and hemianopic vision. Proc Biol Sci (2000) 267:431–8.
Wallace MT, Stein BE. Cross-modal synthesis in the midbrain depends on input from cortex. J Neurophysiol (1994) 71:429–32.
Wallace MT, Wilkinson LK, Stein BE. Representation and integration of multiple sensory inputs in primate superior colliculus. J Neurophysiol (1996) 76:1246–66.
Wallace MT, Roberson GE, Hairston WD, Stein BE, Vaughan JW, Schirillo JA. Unifying multisensory signals across time and space. Exp Brain Res (2004) 158:252–8.[Medline]
Warren DH, Welch RB, McCarthy TJ. The role of visual-auditory "compellingness" in the ventriloquism effect: implications for transitivity among the spatial senses. Percept Psychophys (1981) 30:557–64.[Web of Science][Medline]
Weiskrantz L. Blindsight: a case study and implications (1986) Oxford: Clarendon Press; Oxford University Press.
Welch RB, Warren DH. Immediate perceptual response to intersensory discrepancy. Psychol Bull (1980) 88:638–67.[CrossRef][Web of Science][Medline]
Wilkinson LK, Meredith MA, Stein BE. The role of anterior ectosylvian cortex in cross-modality orientation and approach behavior. Exp Brain Res (1996) 112:1–10.[Web of Science][Medline]
Zeki S, Ffytche DH. The Riddoch syndrome: insights into the neurobiology of conscious vision. Brain (1998) 121(Pt 1):25–45.
Zihl J, Kennard C. Disorders of higher visual functions. In: Neurological disorders: course and treatment—Brandt T, Caplan LR, Dichgans J, Diener HC, Kennard C, eds. (1996) San Diego: Academic Press. 201–12.
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||