Abstract
Speech and gesture are two integrated and temporally coordinated systems. Manual gestures can help second language (L2) speakers with vocabulary learning and word retrieval. However, it is still under-investigated whether the synchronisation of speech and gesture has a role in hel** listeners compensate for the difficulties in processing L2 aural information. In this paper, we tested, in two behavioural experiments, how L2 speakers process speech and gesture asynchronies in comparison to native speakers (L1). L2 speakers responded significantly faster when gestures and the semantic relevant speech were synchronous than asynchronous. They responded significantly slower than L1 speakers regardless of speech/gesture synchronisation. On the other hand, L1 speakers did not show a significant difference between asynchronous and synchronous integration of gestures and speech. We conclude that gesture-speech asynchrony affects L2 speakers more than L1 speakers.
Similar content being viewed by others
Data Availability
Experiment 1 stimuli can be found at doi: https://doi.org/10.6084/m9.figshare.13421336.
Experiment 2 stimuli are available at doi: https://doi.org/10.6084/m9.figshare.13421495.
The data sets of experiment 1 and 2 are available at doi: https://doi.org/10.6084/m9.figshare.13377161.
Notes
Videos for experiment 1 are at DOI: https://doi.org/10.6084/m9.figshare.13421336.
The Common European Framework of Reference for Languages (CEFR) is a guideline used to describe foreign language learners’ language proficiency. It has six reference levels: from A1 for beginners to C2 for proficient learners. B2 and C1 correspond to upper-intermediate to low-advanced, respectively.
The data sets for experiments 1 and 2 can be found at DOI: https://doi.org/10.6084/m9.figshare.13377161.
The videos for experiment 2 are at DOI: https://doi.org/10.6084/m9.figshare.13421495.
References
Alibali, M. W., Kita, S., & Young, A. J. (2000). Gesture and the process of speech production: We think, therefore we gesture. Language and Cognitive Processes, 15, 593–613.
Allen, L. Q. (1995). The effects of emblematic gestures on the development and access of mental representations of french expressions. Modern Language Journal, 79, 521–529.
Anikin, A., Nirme, J., Alomari, S., Bonnevier, J., & Haake, M. (2015). Compensation for a large gesture-speech asynchrony in instructional videos. In G. Ferré & M. Tutton (Eds.), Gesture and Speech in Interaction – 4th edition (GESPIN 4, pp. 19–23).
Bäckman, L., & Nilsson, L. G. (1984). Aging effects in free recall: An exception to the rule. Human Learning, 3, 53–69.
Baroni, M., Bernardini, S., Comastri, F., Piccioni, L., Volpi, A., Aston, G. (2004). Introducing the la Repubblica corpus: a large, annotated, TEI(XML)-compliant corpus of newspaper Italian. In Proceedings of LREC 2004 (pp. 1771–1774). ELDA.
Barr, D. J., Levy, R., Scheepers, C., & Tily, H. J. (2013). Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language, 68, 255–278.
Bates, D., Maechler, M., Bolker, B., & Walker, S. (2014). lme4: Linear mixed-effects models using Eigen and S4 (R package version 1.1-7) [Statistical software]. Available: CRAN.R-project.org/package = lme4.
Beattie, G., & Shovelton, H. (2000). Iconic hand gestures and the predictability of words in context in spontaneous speech. British Journal of Psychology, 91, 473–492.
Beattie, G., & Coughlan, J. (1999). An experimental investigation of the role of iconic gestures in lexical access using the tip-of-the-tongue phenomenon. British Journal of Psychology, 90, 35–56.
Butterworth, B., & Beattie, G. (1978). Gesture and silence as indicators of planning in speech. In R. Campbell, & P. T. Smith (Eds.), Recent advances in the psychology of language: Formal and experimental approaches (pp. 347–360). Plenum.
Butterworth, B., & Hadar, U. (1989). Gesture, speech, and computational stages: A reply to McNeill. Psychological Review, 96, 168–174.
Campbell, R., & Dodd, B. (1980). Hearing by eye. Quarterly Journal of Experimental Psychology, 32, 85–99.
Chui, K. (2005). Temporal patterning of speech and iconic gestures in conversational discourse. Journal of Pragmatics, 37, 871–887.
Cohen, R. L. (1981). On the generality of some memory laws. Scandinavian Journal of Psychology, 22, 267–281.
Cohen, R. L., & Stewart, M. (1982). How to avoid developmental effects in free recall. Scandinavian Journal of Psychology, 23, 9–15.
Cohen, R. L. (1983). The effect of encoding variables on the free recall of words and action events. Memory & Cognition, 11, 575–582.
Cohen, R. L., Sandler, S. P., & Schroeder, K. (1987). Aging and memory for words and action events: Effects of item repetition and list length. Psychology and Aging, 2, 280–285.
Cook, S. W., Yip, T. K., & Goldin-Meadow, S. (2010). Gesturing makes memories that last. Journal of Memory and Language, 63, 465–475.
Davies, M. (2013). Corpus of News on the Web (NOW): 3 + billion words from 20 countries, updated every day. Available online at https://corpus.byu.edu/now/.
de Ruiter, J. P. (2000). The production of gesture and speech. In D. McNeill (Ed.), Language and gesture (pp. 284–311). Cambridge University Press.
Engelkamp, J., & Krumnacker, H. (1980). Imaginale und motorische Prozesse beim Behalten verbalen Materials Zeitschrift für Experimentelle und Angewandte Psychologie, 27, 511–533.
Ekman, P., & Friesen, W. V. (1969). The repertoire of Nonverbal Behavior: Categories, Origins, usage, and Coding (1 vol., pp. 49–98). Semiotica.
Esteve-Gibert, N., & Prieto, P. (2014). Infants temporally coordinate gesture–speech combinations before they produce their first words. Speech Communication, 57, 301–316.
Esteve-Gibert, N., & Prieto, P. (2015). Nine-month-old infants are sensitive to the temporal alignment of prosodic and gesture prominences. Infant Behaviour & Development, 38, 126–129.
Feyereisen, P. (2009). Enactment effects and integration processes in younger and older adults’ memory for actions. Memory (Hove, England), 17, 374–385.
Frick-Horbury, D., & Guttentag, R. E. (1998). The effects of restricting hand gesture production on lexical retrieval and free recall. The American Journal of Psychology, 111, 43–62.
Frick-Horbury, D. (2002a). The effects of hand gestures on verbal recall as a function of high- and-low-verbal-skill levels. Journal of General Psychology, 129, 137–147.
Frick-Horbury, D. (2002b). The use of hand gestures as self-generated cues for recall of verbally associated targets. The American Journal of Psychology, 115, 1–20.
Giorgolo, G., & Verstraten, F. A. J. (2008). Perception of speech-and-gesture integration. In AVSP-2008, International Conference on Auditory-Visual Speech Processing (pp. 31 – 6).
Graziano, M., & Gullberg, M. (2018). When speech stops, gesture stops: Evidence from cross-linguistic and developmental comparisons. Frontiers in Psychology, 9, 879.
Graziano, M., Nicoladis, E., & Marentette, P. (2020). How referential gestures align with speech: Evidence from monolingual and bilingual speakers. Language Learning, 70, 266–304.
Gullberg, M. (2006). Handling discourse: Gestures, reference Tracking, and communication strategies in early L2. Language Learning, 56, 155–196.
Habets, B., Kita, S., Shao, Z., Özyurek, A., & Hagoort, P. (2011). The role of synchrony and ambiguity in speech-gesture integration during comprehension. Journal of Cognitive Neuroscience, 23, 1845–1854.
Helstrup, T. (1984). Serial position phenomena: Memory for acts, contents and spatial position patterns. Scandinavian Journal of Psychology, 25, 131–146.
Hopp, H. (2010). Ultimate attainment in L2 inflection: Performance similarities between non-native and native speakers. Lingua, 120, 901–931.
Kelly, S. D., McDevitt, T., & Esch, M. (2009). Brief training with co-speech gesture lends a hand to word learning in a foreign language. Language and Cognitive Processing, 24, 313–334.
Kendon, A. (1980). Gesture and speech: Two aspects of the process of utterance. In M. R. Key (Ed.), Nonverbal communication and language (pp. 207 – 27). Mouton.
Kendon, A. (2000). Language and gesture. Unity or duality. In D. McNeill (Ed.), Language and gesture (pp. 47–63). Cambridge University Press.
Kendon, A. (2004). Gesture: Visible action as utterance. Cambridge University Press.
Kirchhof, C. (2014). Desynchronized speech-gesture signals still get the message across. The 7th International Conference on Multimodality (7ICOM), Hongkong. Retrieved from: https://pub.uni-bielefeld.de/download/2522747/2901722/ISGS_Caro_presented.pdf
Kita, S. (2000). How representational gestures help speaking. In D. McNeill (Ed.), Language and gesture (pp. 162–185). Cambridge University Press.
Kita, S., & Ozyurek, A. (2003). What does cross-linguistic variation in semantic coordination of speech and gesture reveal? Evidence for an interface representation of spatial thinking and speaking. Journal of Memory and Language, 48, 16–32.
Krauss, R. M., & Hadar, U. (1999). The role of speech-related arm/hand gestures in word retrieval. In L. S. Messing & R. Campbell (Eds.), Gesture, speech, and sign (pp. 93–116). Oxford University Press. https://doi.org/10.1093/acprof:oso/9780198524519.003.0006
Lantolf, J. P. (2010). Minding your hands: The function of gesture in L2 learning. In R. Batstone (Ed.), Sociocognitive perspectives on language use and language learning (pp. 131–150). Oxford University Press.
Loehr, D. (2007). Aspects of rhythm in gesture and speech. Gesture, 7, 179–214.
Lopez-Ozieblo, R. (2019). Cut-offs and co-occurring gestures: Similarities between speakers’ first and second languages.IRAL -. International Review of Applied Linguistics in Language Teaching, 4, 219–265.
Macedonia, M., & Knösche, T. R. (2011). Body in Mind: How Gestures Empower Foreign Language Learning. Mind, Brain, and Education, 5, 196–211.
McCafferty, S. G. (1998). Nonverbal expression and L2 private speech. Applied Linguistics, 19, 73–96.
McDonald, J. L. (2006). Beyond the critical period: Processing-based explanations for poor grammaticality judgment performance by late second language learners. Journal of Memory and Language, 55, 381–401.
McNeill, D. (1985). So you think gestures are nonverbal? Psychological Review, 92, 271–295.
McNeill, D. (1992). Hand and mind: What gestures reveal about thought. University of Chicago Press.
McNeill, D. (Ed.). (2000).Language and gesture.Cambridge University Press.
McNeill, D. (2015). Why we gesture. Cambridge University Press.
McNeill, D., & Duncan, S. D. (2000). Growth points in thinking-for-speaking. In D. McNeill (Ed.), Language and gesture (pp. 141–161). Cambridge University Press.
Morett, L. M. (2018). Hand and in mind: Effects of gesture production and viewing on second language word learning. Applied Psycholinguistics, 39, 355–381.
Mulligan, N. W., & Hornstein, S. L. (2003). Memory for actions: Self-performed tasks and the reenactment effect. Memory & Cognition, 31, 412–421.
Navarra, J., Alsius, A., Soto-Faraco, S., & Spence, C. (2010). Assessing the role of attention in the audio-visual integration of speech. Information Fusion, 11, 4–11.
Nobe, S. (1993). Cognitive processes of speaking and gesturing: A comparison between first language speakers and foreign language speakers Masters thesis, Dept. of Psychology, University of Chicago.
Obermeier, C., Holle, H., & Gunter, T. C. (2011). What iconic gesture fragments reveal about Gesture–Speech Integration: When Synchrony is lost, memory can help. Journal of Cognitive Neuroscience, 23, 1648–1663.
Obermeier, C., & Gunter, T. C. (2015). Multisensory integration: The case of a time window of gesture-speech integration. Journal of Cognitive Neuroscience, 27, 292–307.
Peirce, J. W., Gray, J. R., Simpson, S., MacAskill, M. R., Höchenberger, R., Sogo, H., Kastman, E., & Lindeløv, J. (2019). PsychoPy2: Experiments in behavior made easy. Behavioral Reserch Method, 51, 195–203.
Saltz, E., & Donnenwerthnolan, S. E. (1981). Does motoric imagery facilitate memory for sentences? A selective interference test. Journal of Verbal Learning and Verbal Behaviour, 20, 322–332.
Segalowitz, N. S. (2003). Automaticity and second language learning. In C. Doughty, & M. Long (Eds.), The handbook of second language acquisition (pp. 382–408). Blackwell.
Seyfeddinipur, M. (2006). Disfluency: Interrupting speech and gesture, Ph.D. thesis, Radboud University Nijmegen, Nijmegen.
Smotrova, T., & Lantolf, J. P. (2013). The function of gesture in Lexically focused L2 instructional conversations. Modern Language Journal, 97, 397–416.
Sweller, N., Shinooka-Phelan, A., & Austin, E. (2020). The effects of observing and producing gestures on japanese word learning. Acta Psychologica, 207, 1039.
Tellier, M. (2005). How do teacher’s gestures help young children in second language acquisition? Université Paris 7, UFR Linguistique, Laboratoire ARP.
Tellier, M. (2008). The effect of gestures on second language memorisation by young children. Gesture, 8, 219–235.
Treffner, P., Peter, M., & Kleidon, M. (2008). Gestures and phases: The dynamics of speech-hand communication. Ecological Psychology, 20, 32–64.
van Compernolle, R. A., & Williams, L. (2011). Thinking with your hands: Speech–gesture activity during an L2 awareness-raising task. Language Awareness, 20, 203–219.
Wesp, R., Hesse, J., Keutmann, D., & Wheaton, K. (2001). Gestures maintain spatial imagery. The American Journal of Psychology, 144, 591–600.
Westfall, J., Kenny, D. A., & Judd, C. M. (2014). Statistical power and optimal design in experiments in which samples of participants respond to samples of stimuli. Journal of Experimental Psychology: General, 143, 2020–2045.
Yasinnik, Y., Shattuck-Hufnagel, S., & Veilleux, N. (2005). Gesture marking of disfluencies in spontaneous speech. Retrieved from http://www.isca-speech.org/archive_open/archive_papers/diss_05/dis5_173.pdf
Acknowledgements
The authors thank Andrea Liotto for videorecording and editing the stimuli, as well as hel** collect the data of Experiment 2. We also thank Debora Lupini for her help with the data collection of Experiment 1.
Author information
Authors and Affiliations
Contributions
Author 1 and author 2 have contributed equally to the research and the final manuscript.
Corresponding author
Ethics declarations
Ethical Approval
Informed consent to participate in the experiment was obtained from each participant included in the study. The informed consent module subscribed by the participants was approved by the Ethical Committee for Research at the Università degli Studi di Padova (http://ethos.psy.unipd.it/it/composizione/) and can be found at the following link: http://ethos.psy.unipd.it/it/documentazione/. All procedures performed in the studies involving human participants followed the institutional and national research committee’s ethical standards and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. For further information, see: http://ethos.psy.unipd.it/it/collegamenti/.
Conflict of Interest/Competing interests -Funding
The research has not been funded.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Cavicchio, F., Busà, M. The Role of Representational Gestures and Speech Synchronicity in Auditory Input by L2 and L1 Speakers. J Psycholinguist Res 52, 1721–1735 (2023). https://doi.org/10.1007/s10936-023-09947-2
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10936-023-09947-2