Ecological acoustics and the multi-modal perception of rooms: Real and unreal experiences of auditory-visual virtual environments

Abstract

An ecological approach to multimodal perception of virtual environments suggests that different perceptual mechanisms should cooperate in forming an impression of the complex surrounding. Traditionally, Virtual Environments (VE's) has primarily been developed for the visual modality. It is hypothesized that multi-modal stimulation in VE's raises the experience of presence perceived by the user. Furthermore, it is believed that auditory cues also can improve memory. In Experiment 1, 40 subjects were assigned either to a unimodal (vision only) or bimodal (vision and hearing) virtual environment. The subjects had two memory- and navigation tasks, one where auditory cues had no apparent connection to visual information and one where auditory and visual cues carried similar information. Completion time for both tasks was measured. Statistical analysis showed as expected that no improvement of memory occurred for the unrelated task, while the auditory information yielded a significant effect in the second memory task. Ratings showed that subjects in the bimodal condition experienced significantly higher presence, were more focused on the situation and enjoyed the VE more than subjects receiving unimodal information did. Experiment 2 tested the hypothesis that varying degrees of visual realism would affect judgments of aural room qualities in a betweensubjects design using 80 undergraduates. The results suggested that auditory stimuli in virtual environments can serve both as an information-carrying channel as well as way to improve the experience of presence in a VE and that memory performance may serve as a measure of presence in VE's.