The Efficiency of Augmented Pointing with and Without Speech in a Collaborative Virtual Environment

Abstract

Pointing is a ubiquitous part of human communication. However, pointing gestures to distal referents are often misunderstood systematically, which may limit the usefulness of pointing. We examined pointing-based communication in a collaborative virtual environment (CVE) to address three questions. First, we wanted to evaluate the potential of apparently natural but technically augmented pointing in CVEs, such as presenting a warped pointer for increased legibility or the ability to assume the pointer’s perspective. Second, we wanted to test whether technical improvements in pointing accuracy also facilitate communication if pointing is accompanied by speech. Third, we wanted to check whether pointing accuracy is correlated with the efficiency of communication involving pointing and speech. An experiment revealed that pointing-based communication has considerable potential to be enhanced in CVEs, albeit the specific augmentation procedure we employed did not improve pointing-based communication. Importantly, improvements in pointing accuracy also facilitated communication when speech was allowed. Thereby, speech reduced but could not rule out misunderstandings. Finally, even a small gain in pointing accuracy allowed participants to agree on a referent faster. In summary, the experiment suggests that augmented pointing may considerably improve interactions in CVEs. Moreover, speech cannot fully compensate misunderstandings of pointing gestures and relatively small differences in pointing accuracy affect the efficiency of communication of speech is allowed.