Perceived synchrony in a bimodal display: Optimal intermodal delay for coordinated auditory and haptic reproduction

Abstract

The purpose of this study was to determine the range of optimum intermodal delay values for coordinated auditory and haptic reproduction of brief impact events. Indirect psychophysical methods were used to find the intermodal delay that would be most likely to generate the response of perceived synchrony between acoustic and structural vibration components of those events. A recording of a representative impact sound was processed to create bimodal stimuli with varying amounts of intermodal delay between the bimodally reproduced components. The haptic component of the bimodal stimulus was whole-body vibration presented via a platform on which the observer was seated. Using four actuators moving together, users could be displaced linearly upwards or downwards, with a very quick response and with considerable force (the feedback-corrected linear system frequency response was flat to 50 Hz). The auditory component of the bimodal stimulus was presented in an immersive virtual acoustic environment via a multichannel reproduction of simulated indirect sound. The direct sound component matched to the haptic stimulus was reproduced via a frontally-located pair of loudspeakers that included a low-frequency driver capable of reproducing sound with a linear frequency response ranging from 25 to 300 Hz and a high-frequency driver extending well above 20kHz. The intermodal delay was adaptively varied using a two-alternative, forced-choice (2AFC) procedure to track the point of subjective simultaneity (PSS) based upon temporal order judgments with the following response options: 1) haptic sensation seemed to precede auditory sensation; and 2) haptic sensation seemed to follow auditory sensation. Then, in order to avoid sequential response biases in the tracking procedure, a constant stimulus method was used to determine directly the optimal range of intermodal delay values for producing observer responses of intermodal synchrony, with two response options: haptic sensation either seemed to precede or to follow auditory sensation.