Integration of sensory information within touch and across modalities

Transcription

1 Integraton of sensory nformaton wthn touch and across modaltes Marc O. Ernst, Jean-Perre Brescan, Knut Drewng & Henrch H. Bülthoff Max Planck Insttute for Bologcal Cybernetcs Tübngen, Germany Abstract We perceve the world surroundng us va multple sensory modaltes, ncludng touch, vson and audton. The nformaton derved from all these dfferent modaltes has to converge n order to form a coherent and robust percept of the world. Here, we revew a model (the MLE model) that n the statstcal sense descrbes an optmal ntegraton mechansm. The beneft from ntegratng sensory nformaton comes from a reducton n varance of the fnal perceptual estmate. We here llustrate ths ntegraton mechansm n the human bran wth two examples: the fst example demonstrates the ntegraton of force and poston cues to shape wthn haptc percepton; the second example hghlghts multmodal percepton and shows that tactle and audtory nformaton for temporal percepton nteracts n a way predcted by the MLE ntegraton model. Haptc Percepton; Multmodal Interactons; Cue Integraton; Maxmum-Lkelhood Estmaton. I. INTRODUCTION A. Integraton of Sensory Informaton Our bran s constantly fed wth a contnuous stream of sensory nformaton acqured from all the dfferent sensory modaltes. The resultng multmodal percept has to be coherent and unambguous to enable nteractons wth the envronment. But how does the bran come up wth such a unque percept? To llustrate the problem, magne drvng a nal nto wood usng a hammer. The poston of the nal n space can be seen, but may also be felt, whle holdng the nal n one hand. That s, vson and touch now provde redundant nformaton about the nals poston n space. To accurately ht the nal wth the hammer the poston nformaton from the two modaltes must be ntegrated nto one common representaton. Slght dscrepances n the representaton of nformaton between the modaltes, as they naturally arse due to the probablstc nature of sensory estmaton (cf. next secton), results n an nterestng stuaton: the observer ether has to decde whch nformaton (modalty) to trust n a gven stuaton (vson or touch) or t has to fnd a way to best combne the dscrepant nformaton to come to an optmal decson (or acton). On the other hand, havng more than one (redundant) estmate avalable can be an advantage: The accuracy wth whch an envronmental property can be judged ncreases wth the number of ndvdual perceptual estmates avalable. In the hammer example, the poston n space can be estmated more relably usng both modaltes (vson and touch) nstead of only one. One could speculate that ths may be one reason why t s better for you to hold the nal yourself, nstead of havng someone else hold t for you whle hdng t wth the hammer. Not all nformaton derved from dfferent modaltes s redundant. In the majorty of cases nformaton derved from the dfferent modaltes wll be complementary n nature, such as when feelng an object s weght whle seeng ts color. Naturally, dfferent combnaton rules have to be appled for combnng such complementary nformaton nto a stable percept (cf. [1] for a recent revew). Here, we concentrate on the ntegraton mechansms for redundant sensory nformaton, such as that for a spatal poston, that can be seen and felt, that for haptc shape that can be derved from force and poston cues (see Chapter II), or that for a number of sequentally perceved events that can be seen, felt, or heard (see Chapter III). B. The probablstc nature of sensory estmaton The problem of sensory combnaton can be understood usng sgnal detecton theory [2]. Percepton s a probablstc process. If one estmates some envronmental property, such as an object s sze, ths estmate wll have a varance assocated wth t. As a result, f the same envronmental property s estmated consecutvely 100 tmes, all 100 perceptual estmates may vary slghtly. Fgure 1 shows schematcally the probablty densty functon for the estmaton of an object s sze s. In the smplest case ths probablty densty functon has a Gaussan shape and s unbased. Ths curve s then defned by ts mean S, whch for an unbased estmator corresponds to the objects sze, and ts standard devaton : ˆ S = N(S, ). If we defne the relablty r as the nverse of the varance 2 : r = 1 2,

2 then the larger the varance (or standard devaton) of the assocated dstrbuton the less relably s the assocated perceptual estmate. robust, because when there s one estmate not avalable at a gven tme (or ts nformaton s degraded) the other estmate can substtute for t. The second advantage s that the fnal estmate becomes potentally more relable compared wth the relablty of the ndvdual estmates feedng the combned percept. What would be the statstcally optmal strategy for combnng redundant sensory nformaton? Fgure 2 shows the probablty densty functons for two ndependent estmates derved from two dfferent modaltes. In the example dscussed here t s a sze estmate that s derved from the vsual and haptc modaltes ( S ˆ V and S ˆ H ). Accordng to the Maxmum-Lkelhood-Estmaton (MLE) scheme the ntegrated sze estmate S ˆ VH s a weghted average across the ndvdual sensory sgnals wth weghts w that sum up to unty (the ndces, j refer to the ndvdual modaltes) [3]. S ˆ = w ˆ S wth w =1. Fgure 1. A) Schematc llustraton of the probablty densty functon for the estmaton of an object s sze s. The hstogram ndcates the dstrbuton of answers derved from the sze estmaton process. The ftted curve has a Gaussan shape (wth standard devaton and mean S ) and ndcates the probablty densty functon. B) Schematc drawng of a psychometrc functon derved usng a 2-nterval forced-choce task gven the probablty densty functon for estmatng the object s sze from A. The just notceable dfference (JND) derved at the 0.84 pont corresponds to JND = 2. PSE s the pont of subjectve equalty. How can the varance or the relablty of a sensory sgnal be estmated expermentally? For ths, classcal psychophyscal dscrmnaton paradgms such as a 2- nterval forced-choce (2-IFC) task can be used. Usng the 2-IFC task subjects have to compare, for example, the szes of two objects presented sequentally (a standard object of sze S 0 and a comparson stmulus of sze S). If the dfference n sze between the two ntervals (S-S 0 ) s large, subjects wll have no problem dscrmnatng them, and consequently they wll make only a few errors. Wth decreasng sze dfference however the error rate wll rse. If the probablty densty functons for S and S 0 are Gaussan wth dentcal varance 2, the resultng psychometrc functon s a cumulatve Gaussan (see Fg. 1). The Just Notceable Dfference (JND) defned at the 84% level (the dfference between the 50% and the 84% ponts) provdes an estmate Optmally, weghts are chosen to be proportonal to the relablty of a gven sgnal. That s, f the vsual modalty provdes the more relable nformaton n a gven stuaton ths sgnal s weghted hgher n combnaton. w j = r j. r =1.., j,...n In the example shown n Fgure 2 the varance assocated wth the vsual estmate s less than the varance assocated wth the haptc estmate. That s, the vsual nformaton s more relable. Therefore, the combned estmate beng the weghted sum s closer to the vsual than the haptc estmate. Under other crcumstances where the haptc modalty provdes the more relable estmate the stuaton s reversed. JND = 2 for the varablty of the underlyng Gaussan dstrbuton. C. The MLE model for sensory ntegraton Redundant sgnals may to some degree sound lke a waste of nformaton. But actually ths s not necessarly so. There are two major advantages n havng avalable redundant nformaton: the frst s that the system s more Fgure 2. Schematc drawng of the probablty densty functons of the ndvdual vsual and haptc estmates and of the combned vsual-haptc estmate, whch s a weghted average accordng to the MLE ntegraton rule. The varance assocated wth the vsual-haptc dstrbuton s less than ether of the two ndvdual estmates. The varance of the combned estmate wll be less then that of ether of both ndvdual estmates feedng the

3 combnaton process. In other words the relablty can only mprove by combnng nformaton (provded the sgnals and ther nose dstrbutons are ndependent, so there s no nteracton). Followng the MLE prncple the relablty of the combned estmate s the sum of the relabltes of the ndvdual estmates: r = r One can show that the MLE ntegraton scheme s statstcally optmal n that t provdes the most relable unbased sensory estmate, gven that the ndvdual estmates are Gaussan dstrbuted and that these nose dstrbutons are ndependent. However, even f the nose dstrbutons of the ndvdual estmates show some correlaton one can stll beneft from combnng sensory nformaton and the combned estmate may become more relable than the ndvdual estmates [4]. In a recent study we could show that humans actually ntegrate vsual and haptc sze nformaton n such a statstcally optmal fashon [5]. Others now showed that ths fndng of optmalty holds not only for the ntegraton across vson and touch, but also for the ntegraton of nformaton across and wthn other modaltes, such as audton or vson [6, 7, 8]. Ths suggests that maxmumlkelhood-estmaton s an effectve and wdely used strategy exploted by the perceptual system. In the followng two chapters we llustrate some of the mechansms predcted by the MLE ntegraton scheme. II. WITHIN-HAPTIC CUE INTEGRATION A. Cues to shape n actve touch So far, the MLE model on cue ntegraton has been well supported for wthn-vsual and for crossmodal ntegraton. Wthn the haptc modalty, however, the ntegraton of dfferent cues s yet hardly examned n a correspondng systematc fashon. Whether the MLE model generalze to wthn-haptc ntegraton s partcularly nterestng for the case of actve touch. Ths s because durng actve touch the observer s able to actvely pck up the nformaton that s most mportant for hm. Clearly ths dstngushes actve touch from the less dynamc stuatons so far examned where the observer ntegrates the stmulaton that he/she obtans mostly passvely. Ths seres of experments ams at systematcally explorng cue ntegraton durng actve touch. For ths, a recent fndng from haptc curvature percepton provdes a good startng pont. Robles-de-la-Torre and Hayward [9] dstngushed between postonal and force cues n the percepton of small-scale bumps (ampltude 3 mm): When sldng a fnger across a bump on a surface, the fnger follows the geometry of the bump provdng postonal cues for the shape. At the same tme the fnger s exposed to forces related to the slope of the bump (force cues). In a psychophyscal experment, partcpants moved a probe over a physcal bump, hole or plane; a custom-made devce allowed to manpulate the forces opposng/supportng the movement. Nearly all partcpants reported to feel the shape ndcated by the force cues and not by the postonal cues. Ths fndng tends to ndcate absolute domnance of force over poston cues. However, the task n the experment descrbed by Robles-de-la-Torre and Hayward was to categorze perceved objects n bumps or holes, so that the observed domnance may just ndcate a relatve domnance of one cue over the other whch manfests tself as an absolute domnance n the classfcaton responses gven. Ths s n contrast to an absolute perceptual domnance of force over poston cues. In Experment 1 we ndependently and systematcally vared force and poston cues to haptcally perceved curvature of 3D-arches and quantfed the percept by psychophyscal methods. For that purpose, we constructed a set of vrtual arches wth conflctng cues, where we ntermxed force and poston cues related to curvatures between 0 and 16 /m. Partcpants compared these conflctng-cue shapes to shapes wth cues consstent. B. Experment 1 Integratng force and poston cues 1) Method The haptc stmul were vrtual arches of 30 mm wdth and 50 mm depth that were curved n heght along the depth axs. A PHANToM 1.5 haptc devce (SensAble Technologes) dsplayed the stmul to the rght ndex fnger of the partcpants. Ths devce dsplays vrtual objects by measurng the poston of the fnger and, then, exertng approprate reacton forces on t. We defned the postonal cues of curvature by the postons of the onsets of the object s reacton forces and the force cues as the drectons of these reacton forces. In natural objects reacton forces are normal to the surface and, thus, force and poston cues are hghly correlated. However, wth the PHANToM devce we were able to dsentangle these cues and combne force and poston cues from dfferent arches (Fg. 3). Fgure 3. Force and poston cues n arches and ther combnaton We constructed nne standard stmul n whch we completely ntermxed force and poston cues to curvatures of 0, 8, and 16 /m. To each standard curve belonged a set of 13 non-conflctng comparson curves, n each of whch force and poston cues ndcated the same curvature. Curvatures of the comparson stmul were dstrbuted n a range of +/-9/m around the expected pont of subjectve equalty (PSE) for the correspondng standard curve,.e. around that comparson curve whch we expected to be ndstngushable from the standard curve. Ths work was supported by the Max Planck Socety and by the 5 th Framework IST Program of the EU (IST , TOUCH-HapSys).

4 Usng a 2-IFC task each tral n the experment conssted of the sequental presentaton of a standard and a comparson. Then, partcpants had to decde whch of the two stmul was more convex. We dsplayed each par of stmul 16 tmes to each partcpant followng the method of constant stmul. The order of stmul wthn the trals was balanced, the order of the pars completely randomzed. The entre experment lasted about 8 hours that were dvded nto four sessons accomplshed on dfferent days. Seven rght-handers took part n the experment - naïve to the purpose of the study and pad to partcpate. In the data analyss we determned for each partcpant and each standard stmulus the psychometrc functon, that s the proporton of trals n whch the comparson was perceved as more convex than the standard aganst the curvature of the comparson. Fts were done wth a cumulatve Gaussan functon usng the psgnft toolbox verson for Matlab whch mplements the maxmum-lkelhood fttng method descrbed n [10]. From these fts, we obtaned ndvdual PSEs. 2) Results & Dscusson Indvdual PSEs (Fg. 4) entered nto an ANOVA wth the wthn-partcpant varables Poston Curvature (0, 8, 16/m) and Force Curvature (0, 8, 16 /m). Both man effects were relable, (Poston Curvature, F(2,12)=228.8, p<.001; Force Curvature, F(2,12)=80.4, p<.001), ndcatng that PSEs systematcally ncreased wth ncreasng curvature specfed by ether cue. So, both force and poston cues contrbuted to perceved curvature, whch s consstent wth the MLE model. Fgure 4. Average PSEs and ther standard errors n Experment 1 The MLE model further predcts that the combnaton of two cues can be descrbed by a weghted lnear combnaton. Here, we found a margnally sgnfcant nteracton between Force Curvature and Poston Curvature, F(4,24)=3.6, p<.05. Ths ponts to some volaton of lnearty. However, the lnearty assumpton s only true f the relablty of the sgnal does not change wth curvature. Further, a multple lnear regresson (ntercept = 0) of the average PSEs on the curvatures ndcated by poston and force cues explaned the varance between PSEs to 99%, ndcatng that weghted averagng s at least a good approxmaton. The relatve weght for the force cue was 46% and for the poston cue 54%. Fnally, ndvdual multple regressons confrm that each person combnes force and poston cues approxmately lnearly (varance explaned > 94%) and that, also ndvdually, both cues (force weghts: 28% to 61%) contrbute to perceved curvature. Beng questoned after the experment none of the partcpants reported to have notced the conflcts between the force and poston cues. Taken together, fndngs from ths experment suggest that haptcally perceved curvature s a weghted average of both postonal and force cues. Ths s consstent wth the MLE model on cue ntegraton. Another predcton of the MLE model s that the cue weghtngs depend on the relatve relablty of the respectve cue. The margnal nteracton we found between the two cues may hnt at a change of weght wth curvature (.e., the sze of the aches) Experment 2 provded a frst test of ths predcton. In Experment 2, two sets of vrtual shapes realzed shallow vs. hgh arches. Keepng the length of the curved secton constant, we ntermxed force and poston cues ndcatng curvatures of 0, 5, and 10/m (shallow arches) and of 20, 25, and 30/m (hgh arches), respectvely. We expected that curvature percepton n hgh arches poston cues have relatvely more weght than n the percepton of shallow arches. We expect ths, because postonal changes of the fngers durng strokng hgh arches are much more pronounced than for shallow ones. C. Experment 2 Changng weght wth sze of arch 1) Methods Apparatus, stmulus constructon and the procedure n sngle trals were the same as n Experment 1. There were three wthn-partcpant varables: Arch Rse (shallow [x = 5/m], and hgh [x=20/m]), Force Curvature (x-5, x, x+5/m), and Poston Curvature (x-5, x, x+5/m) realzed n 18 standard shapes. Each partcpant conducted one double-starcase (1- up/1-down) per standard stmulus (random order), n each of whch two adaptve starcases were nterleaved. Step sze was adjusted from 8/m at the begnnng to 2/m for shallow arches and from 6/m to 1.5/m for hgh arches wthn 3 reversals. Each starcase stopped after 8 nonreducng reversals. From averages of the comparsons curvatures across these 8 reversal ponts, we estmated the PSEs [11]. The experment lasted about 2.5 hours ncludng an ntal short practce phase. We tested 14 pad, naïve partcpants (all rght-handed). 2) Results & Dscusson Indvdual PSEs (Fg. 5) entered an ANOVA wth the varables Arch Rse, Force Curvature, and Poston Curvature. A man effect of Arch Rse, F(1,13)=3420.1, p<.001 confrmed that the hgh arches were generally perceved as beng more convex than the shallow arches. Man effects of Poston Curvature, F(2, 26)=357.1, p<.001, and Force Curvature, F(2, 26)=464.5, p<.001, confrm the fndng from the prevous experment that PSEs ncrease wth ncreasng curvature specfed by ether cue. Most mportantly, nteractons of Arch Rse wth Poston Curvature, F(2, 26)=26.2, p<.01, as well as wth

5 Force Curvature, F(2, 26)=39.2, p<.01, ndcate that the two cues contrbuted dfferently to perceved curvature n the two sets of arches. In other words, the relatve weghtng of the two cues changes wth the sze of the arch. There were no other relable effects (Fs<1). Fgure 5. Average PSEs and ther standard errors n Experment 2 In a second step, we calculated multple regressons (ntercept = 0) of the PSEs on the standards force and poston curvatures separated by Arch Rse: For shallow arches the relatve force cue weght was 67%, the poston cue weght 33% and varance was explaned to 99%; for hgh arches the values were 41% force, 59% poston, and 99% explaned varance. Consstent wth the MLE model, the hgh amount of explaned varance shows for each arch set alone, that perceved curvature can be descrbed by weghted averagng of the curvatures ndcated by the two cues. Cue weghts show that force cues nfluence the percept more n the set of shallow as compared to hgh arches and vce versa for poston cues. Ths s consstent wth our hypothess that the more pronounced the fnger ampltudes the hgher the relatve poston cue weght. Also consstent s the relatve force cue domnance n the shallow arch set that was prevously observed [9]. Most mportantly, the dfference n cue weghts dependng on arch set s consstent wth the predcton of the MLE model that cue weghtngs change wth the relatve relabltes of the respectve cues. Agan, none of the observers reported to have notced the conflcts between the cues. D. Concluson: Wthn haptc cue ntegraton In ths seres of experments we have explored whether prncples of cue ntegraton known from ntegraton wthn and across other modaltes and formulated by the MLE model can be extended to actve haptc percepton. Experment 1 demonstrated that the ntegraton of force and poston cues to haptc shape can be descrbed by a lnear weghtng model. Most mportantly, ths was true for each ndvdual subjects data. Experment 2 corroborated ths assumpton and, further, demonstrated that cue weghts depend on the curvature of the shape. Poston cue weghts were hgher for curvature percepton of hgh as compared to shallow arches - probably medated va changng cue relablty. Taken together, also wthn-haptc cue ntegraton seems to be well descrbed by the MLE model. Thereby, to our knowledge the present seres of experments s the frst systematc study on cue ntegraton durng actve haptc exploraton. III. AUDITORY-TACTILE INTEGRATION A. Combnng multmodal temporal events Our everyday nteractons wth the envronment provde us wth a contnuous stmulaton of our dfferent sensory channels. The central nervous system (CNS) has thus to deal wth a pool of multmodal sgnals provdng nformaton of dfferent nature concernng body/ envronment relatonshp. In many cases, the occurrence of a specfc value of the sgnal n one sensory modalty s accompaned by a correspondng specfc sgnal n one or more other modaltes. For nstance, when knockng on a door, one gets congruent vsual, tactle and audtory feedback, ths feedback beng specfc to the characterstcs of the acton (e.g., number of tmes one knocked, delay n between two knocks, knockng force s ntensty). Several psychophyscal experments suggested that these redundant sensory sgnals are automatcally co-regstered to derve a coherent unfed percept of the presented stmul [12-18]. One convenent way of determnng whether two sgnals are automatcally combned conssts n testng whether a to-be-gnored background sgnal can bas the percepton of a to-be-assessed focal sgnal. In lne wth ths, we tested whether audtory and tactle sgnals are automatcally combned for tactle taps (Experment 3) and audtory beeps (Experment 4) percepton. B. Experment 3 Effect of beeps on taps percepton 1) Methods In the frst experment, we tested whether the percepton of tactle sequences of taps (2 to 4) delvered on the ndex fngertp can be modulated by smultaneously presented sequences of to-be-gnored audtory beeps when the number of beeps dffers (less or more) from the number of taps. Four audtory condtons were assocated to the presentaton of the tactle sequences: No Beep (baselne performance for tactle percepton), One Beep Less (# beeps = # taps-1), Same Amount (# beeps = # taps), and One Beep More (# beeps = # taps+1). Fgure 6. Expermental set-up for tactle audtory stmulaton.

6 The expermental set-up s schematcally represented n Fgure 6. The tactle taps were agan generated va a PHANToM force-feedback devce. A metallc pn of 1 mm n dameter ndented subjects skn by approxmately 2 mm wth a force of 1 N. The subjects could not see ther hand or the force-feedback devce. Fgure 7. Temporal sequence of haptc and audtory events. Haptc taps were 20 ms lang wth ans ISI of 135 ms. Beeps were 50 ms long wth varable ISI (see formula n fgure). Each tap lasted 20 ms and the delay between the onsets of two successve taps was 135 ms (see Fgure 7). The beeps were audtory tones (790 Hz, 74 db) delvered va earphones. Each beep lasted 50 ms and the delay separatng the onsets of two successve beeps vared so that the onsets of the frst and last beeps concded wth the respectve onsets of the frst and last taps. For the whole duraton of the experment, the earphones also emtted a whte-nose (71 db) to mask any external audtory dsturbance. tactle taps [F (3, 69) = 43.66, p <.001]. Post hoc comparsons usng a Sdak adjustment for multple comparsons (p < 0.05) were then performed. Concernng the man effect of the number of taps, each of the 2 (mean = 1.93), 3 (mean = 2.71) and 4 taps (mean = 3.31) condtons were sgnfcantly dfferent from each other. Concernng the man effect of the number of beeps, the No Beep (mean = 2.74) and the Same Amount (mean = 2.69) condtons dd not dffer from one another. The perceved number of taps n the One Beep Less condton (mean = 2.38) was sgnfcantly lower than n any other audtory condton, whereas the perceved number of taps n the One Beep More condton (mean = 2.79) was sgnfcantly hgher than all but the No Beep condton. C. Experment 4 Effect of taps on beeps percepton 1) Methods In the second experment, we tested whether the automatc combnaton of tactle and audtory sgnals also works the other way around,.e., whether the percepton of audtory sequences of beeps (2 to 4) can be modulated by smultaneously presented sequences of to-be-gnored tactle taps when the number of taps dffers (less or more) from the number of beeps. Four tactle condtons were assocated to the presentaton of the tactle sequences: No Tap (baselne performance for audtory percepton), One Tap Less (# taps = # beeps-1), Same Amount (# taps = # beeps), and One Tap More (# taps = # beeps+1). 2) Results Fgure 9. Temporal sequence of haptc and audtory events used n Exp. 4. Each beep lasted 20 ms and the delay between the onsets of two successve beeps was 135 ms (see Fgure 9). Each tap lasted 50 ms and the delay separatng the onsets of two successve taps vared so that the onsets of the frst and last taps concded wth the respectve onsets of the frst and last beeps. Fgure 8. Number of perceved taps as a functon of both the actual number of delvered taps and the audtory condton. The dotted lnes represent subjects average percepton n the No Beep condton As shown n Fgure 8, the perceved number of tactle taps was sgnfcantly (ANOVA) nfluenced by the smultaneous presentaton of to-be-gnored audtory beeps. Indeed, the perceved number of taps not only depended on the actual number of delvered taps [F (2, 46) = , p <.001], but also on the number of smultaneously presented 2) Results The results are presented n the Fgure 10. The perceved number of beeps depended on the actual number of delvered beeps [F (2, 46) = , p <.001], but was also nfluenced by the number of smultaneously presented tactle taps [F (3, 69) = 6.04, p <.01]. Post hoc comparsons usng a Sdak adjustment for multple comparsons (p < 0.05) revealed that each of the 2 (mean = 1.85), 3 (mean = 2.93) and 4 beeps (mean = 3.75) condtons were sgnfcantly dfferent from each other. However, concernng the man effect of the number of taps, only the No Tap (mean = 2.79) and the Same Amount (mean =

7 2.91) condtons sgnfcantly dffered from one another. Nether the One Tap Less (mean = 2.80) nor the One Tap More (mean = 2.87) tactle condton dd dffer from any of the other condtons. Fgure 10. Number of perceved beeps as a functon of both the actual number of delvered beeps and the tactle condton. The Dotted lnes represent subjects average percepton n the No Tap condton D. Dscusson The results of these experments show that audtory and tactle sgnals are automatcally combned both for tactle taps and audtory beeps percepton, and ths even when the subjects are explctly nstructed to gnore the background sgnal. Ths automatc combnaton lkely results from the fact that the matchng between cooccurrng multmodal sgnals s very consstent across our everyday experence, so that the CNS can learn to coregster sets of redundant sensory sgnals and dentfy every sngle set as elcted by the same unque event or stmulus. Because multmodal cues reduce the varance of perceptual estmates [5, 19] and enhance stmulus detecton [20, 21], such automatc combnaton of redundant-lke sensory sgnals can be conceved as an optmzaton process. Our results also show that the combnaton of audtory and tactle sgnals does not lead to smlar results dependng on whether the prmary task s tactle or audtory. Indeed, f audtory background sgnals sgnfcantly based tactle percepton n the frst experment, no real bas of audtory percepton could be evoked by tactle sgnals n the second experment (no effect of the One Tap Less and One Tap More condtons). In ths latter case, the only sgnfcant effect of the taps on audtory percepton was an mproved accuracy of the subjects when redundant audtory and tactle sgnals were avalable ( Same Amount tactle condton) as compared to when only audtory sgnals were provded ( No Tap tactle condton). Ths ndcates that n ths type of non-spatal task (.e., countng the amount of events n gven sequences), there s a relatve domnaton of audtory sgnals on tactle sgnals. In ths respect, our results are n lne wth prevous expermental results reportng audtoryevoked bases of tactle percepton n the non-spatal doman [16-18]. IV. SUMMARY AND CONCLUSION Ths manuscrpt started out wth presentng a model for multmodal sensory ntegraton. Ths model the MLE model descrbes an ntegraton mechansm that n the statstcal sense s optmal. The crteron for optmalty s defned as the precson of the system that s here determned by the relablty (.e., the nverse varance) of the perceptual estmate. We further llustrated detals of the model usng two sets of experments. The purpose of the frst set of experments (Exp. 1 & 2) was to nvestgate whether cues to shape n actve touch ntegrate accordng to the rules of the MLE model. The purpose of the second set of experments (Exp. 3 & 4) was to nvestgate the ntegraton of sound wth tactle nformaton for the percepton of a sequence of short temporal events. From the wthn-haptc cue ntegraton experments we concluded that the MLE model s sutable for descrbng the process of ntegraton of poston and force cues to haptc shape and that the weght gven to each cue changes wth the curvature presented.e., hgh arches have a hgher poston cue weght relatve to small-scale arches, n whch the force cue s weghted relatvely hgher. In practce ths means that the percept of a curvature ndcated by some postonal profle can be nfluenced by the force profle provded along wth the postonal nformaton. In the future such nteractons between force and poston nformaton may be exploted n the renderng of haptc objects and surfaces. For example, one could magne that technques that are well establshed n computer graphcs such as bump or normal mappng can be transferred to the renderng of haptc objects n computer haptcs. Accordng to ths dea the haptc bump or normal map would contan a 2D array of force vectors correspondng to the vrtual normal vector of the surface to be represented. Snce the force cue s the domnant component for small-scale structures, from our basc scentfc studes presented here, we would predct that such cheatng n form of a haptc bump or normal map would work as long as the haptc structures are relatvely small (wthn the range of a few mllmeters). I.e., we propose that on a small scale there s no need to have the postonal nformaton completely correct (accordng to the physcal model) as long as the force profle s calculated approprately. Most obvously the advantage of such a shortcut s the savng of compute power. From the studes on audtory-tactle ntegraton of sequences of short temporal events we could concluded that such multmodal nformaton s automatcally ntegrated, ndependent of whether attenton s put on the one or the other modalty. Such knowledge may now also be systematcally exploted for the haptc renderng of objects and events. For example, one could magne nducng the feelng of two haptc events by playng an accompanyng sound but n realty there was only one haptc event present. Such cheatng may help smplfyng the renderng complexty nherent n some haptc or multmodal applcatons.

8 Taken together, we demonstrated that a very general prncple, such as the MLE model, can be used for explanng many phenomenon n haptc and multmodal ntegraton. Wth ths basc knowledge and ts possblty to make systematc predctons, such phenomenon may now also fnd ts way nto applcatons of haptc or multmodal renderng and dsplay technology. REFERENCES [1] M. O. Ernst, and H. H. Bülthoff, Mergng the senses nto a robust percept, Trends n Cogn. Sc. 8 (4), Aprl 2004, pp [2] D. M. Green and J. A. Swets, Sgnal Detecton Theory and Psychophyscs, Pennsula Publshng, Los Altos, Ca, USA, 1988 [3] W. G. Cochran, Problems arsng n the analyss of a seres of smlar experments, J. Royal Stat. Soc. (Sup.), 4, 1937, pp [4] I. Oruç, T. M. Maloney, and M. S. Landy, Weghted lnear cue combnaton wth possbly correlated error, Vs. Res. 43, 2003, pp [5] M.O. Ernst, and M.S. Banks, Humans ntegrate vsual and haptc nformaton n a statstcally optmal fashon, Nature 415, 2002, pp [6] D. Alas, and D. Burr, The ventrloqust effect results from near optmal crossmodal ntegraton, Curr. Bol. 14 (3), Feb. 2004, pp [7] D. C. Knll, and J. A. Saunders, Do humans optmally ntegrate stereo and texture nformaton for judgments of surface slant? Vs. Res. 43 (24), 2003, pp [8] J. M. Hlls, S. J. Watt, M. S. Landy, and M. S. Banks, Slant from texture and dsparty cues: optmal cue combnaton, Journal of Vson, (submtted). [9] G. Robles-de-la-Torre, and V. Hayward, Force can overcome object geometry n the percepton of shape through actve touch, Nat. 412, 2001, pp [10] F.A. Wchmann, and N.J. Hll, The psychometrc functon: I. Fttng, samplng and goodness-of-ft, Perc. & Psych. 63 (8), 2001, pp [11] J.C. Falmagne, Psychophyscal Measurement and Theory, n Handbook of percepton and Human performance, vol. 2, K.R. Boff, L. Kaufman, and J.P. Thomas, Eds. New York: John Wley & Sons, 1986, pp [12] P. Bertelson, and M. Radeau, Cross-modal bas and perceptual fuson wth audtory-vsual spatal dscordance, Percept. Psychophys. 29, 1981, pp [13] R. Fendrch, and P.M. Corballs, The temporal cross-capture of audton and vson, Percept. Psychophys. 63, 2001, pp [14] S. Moren-Zamr, S. Soto-Faraco, and A. Kngstone, Audtory capture of vson: examnng temporal ventrloqusm, Cog. Bran Res. 17, 2003, pp [15] L. Shams, Y. Kamtan, and S. Shmojo, What you see s what you hear, Nature 408, 2000, p [16] S. Guest, C. Catmur, D. Lloyd, and C. Spence, Audotactle nteractons n roughness percepton, Exp. Bran Res. 146, 2002, pp [17] V. Jousmäk, and R. Har, Parchment-skn lluson: sound-based touch, Curr. Bol. 8, 1998, p [18] K. Höttng, and B. Röder, Hearng cheats touch, but less n congentally blnd than n sghted ndvduals, Psychol. Sc. 15, 2004, pp [19] W.C. Wu, C. Basdogan, and M.A. Srnvasan, Vsual, haptc, and bmodal percepton of sze and stffness n vrtual envronments, P. ASME 67, 1999, pp [20] S.C.A.M. Gelen, R.A. Schmdt, and P.J.M. van den Heuvel, On the nature of ntersensory facltaton of reacton tme, Percept. Psychophys. 34, pp , [21] I.H. Bernsten, M.H. Clark, and B.A. Edelsten, Effects of an audtory sgnal on vsual reacton tme, J. Exp. Psychol. 80, 1969, pp