Abstract
The present study applied the computational tool Coh-Metrix to analyze linguistic and discoursal features of the transcribed interpreting texts from the All China Interpreting Contest (ACIC) in order to predict quality assessments provided by the judges in the ACIC. We find that a stepwise linear regression model with four entries/variables (i.e., word count, lexical diversity, hypernymy of verbs, and frequency of first person singular) could predict 60% of the variance in the human scoring. These results highlight the importance of information completeness in the ACIC, as word count was highly correlated with the human scoring. In addition, the interpreted texts produced by the better performing contestants/interpreters were associated with an increasing level of language sophistication. That is, they tended to use words of higher precision, less frequency, and higher diversity. Based on these results, we argue that even without recourse to diagnostic information related to pronunciation, intonation, rhythm, and speed of delivery, automatically computed analytic indices such as word count, word diversity, and hypernymy of verbs could predict a significant amount of variance in human scoring of interpreting quality. These encouraging, albeit preliminary, findings will prompt us to further explore Coh-Metrix model-based automatic assessment of interpreting quality and also help us gain an enhanced understanding of the “quality” construct perceived by the judges in the ACIC.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Azadnia, Masoud, Ahmad Reza Lotfi, and Reza Biria. 2019. A study of syntactic complexity via Coh-Metrix: Similarities and differences of Ph.D. dissertations written by Iranian University students and English native speakers. RELP 7: 232–254.
Baayen, R. Harald, Richard Piepenbrock, and Leon Gulikers. 1996. The CELEX lexical database (CD-ROM). Philadelphia, PA: Linguistic Data Consortium, University of Pennsylvania.
Bartłomiejczyk, Magdalena. 2007. Interpreting quality as perceived by trainee interpreters: Self-evaluation. Interpreter and Translator Trainer 1 (2): 247–267.
Beals, Diane E. 2002. The CHILDES project: Tools for analyzing Talk: Vol. 1. Transcription format and programs; Vol. 2. The database (3rd ed.). B. MacWhinney. Mahwah, NJ: Erlbaum, 2000. Pp. 366 (Vol. 1); pp. 418 (Vol. 2). Applied Psycholinguistics 23: 304–306. https://doi.org/10.1017/s0142716402222079.
Bühler, Hildegund. 1986. Linguistic (semantic) and extra-linguistic (pragmatic) criteria for the evaluation of conference interpretation and interpreters. Multilingua 5 (4): 231–235.
Cai, **aohong, and Jieyi Zeng. 2004. A review of interpretation quality assessment. Chinese Translators Journal 25 (3): 49–54.
Crossley, Scott, and Danielle McNamara. 2013. Applications of text analysis tools for spoken response grading. Language Learning and Technology 17 (2): 171–192.
Crossley, Scott, Tom Salsbury, and Danielle McNamara. 2009. Measuring l2 lexical growth using hypernymic relationships. Language Learning 59 (2): 307–334.
Crossley, Scott A., and Danielle S. McNamara. 2011. Understanding expert ratings of essay quality: Coh-Metrix analyses of first and second language writing. International Journal of Continuing Engineering Education and Life-Long Learning 21 (2–3): 170–191.
Crossley, Scott A., and Danielle S. McNamara. 2012. Predicting second language writing proficiency: The roles of cohesion and linguistic sophistication. Journal of Research in Reading 35 (2): 115–135.
Crossley, Scott A., Tom Salsbury, and Danielle S. McNamara. 2010. The development of semantic relations in second language speakers: A case for Latent Semantic Analysis. Vigo: International Journal of Applied Linguistics 7 (1): 55–74.
Crossley, Scott A., Tom Salsbury, Danielle S. McNamara, and Scott Jarvis. 2011. What is lexical proficiency? Some answers from computational models of speech data. TESOL Quarterly 45 (1): 182–193.
Field, Andy. 2005. Discovering statistics using SPSS, 2nd ed. Thousand Oaks: Sage.
Flesch, Rudolph. 1948. A new readability yardstick. Journal of Applied Psychology 32 (3): 221–233.
Gallai, Fabrizio. 2016. Point of view in free indirect thought and in community interpreting. Lingua 175–176: 97–121.
Gile, Daniel. 1991. A communication-oriented analysis of quality in nonliterary translation and interpretation. In Translation: Theory and practice, tension and interdependence, ed. Mildred L. Larson, 188–200. Amsterdam: John Benjamins.
Gile, Daniel. 1994. Opening up in interpretation studies. In Translation studies: An interdiscipline, ed. Mary Snell-Hornby, Franz Pöchhacker, and Klaus Kaindl, 149–158. Amsterdam: John Benjamins.
Gile, Daniel. 1995. Basic concepts and models for interpreter and translator training. Amsterdam: John Benjamins.
Gile, Daniel. 2009. Basic concepts and models for interpreter and translator training, Rev ed. Amsterdam: John Benjamins.
González, Luis Pérez. 2006. Interpreting strategic recontextualization cues in the courtroom: Corpus-based insights into the pragmatic force of non-restrictive relative clauses. Journal of Pragmatics 38 (3): 390–417.
Graesser, Arthur C., and Danielle S. McNamara. 2011. Computational analyses of multilevel discourse comprehension. Topics in Cognitive Science 3 (2): 371–398.
Graesser, Arthur C., Danielle S. McNamara, Max M. Louwerse, and Zhiqiang Cai. 2012. Coh-Metrix. http://cohmetrix.com/. Accessed 29 Dec 2020.
Graesser, Arthur C., Danielle S. McNamara, Max M. Louwerse, and Zhiqiang Cai. 2004. Coh-Metrix: Analysis of text on cohesion and language. Behavior Research Methods, Instruments, and Computers 36 (2): 193–202.
Graesser, Arthur C., Murray Singer, and Tom Trabasso. 1994. Constructing inferences during narrative text comprehension. Psychological Review 101 (3): 371–395.
Guo, Liang, Scott A. Crossley, and Danielle S. McNamara. 2013. Predicting human judgments of essay quality in both integrated and independent second language writing samples: A comparison study. Assessing Writing 18 (3): 218–238.
Han, Chao. 2016. Investigating score dependability in English/Chinese interpreter certification performance testing: A generalizability theory approach. Language Assessment Quarterly 13 (3): 186–201.
He, Lianzhen, and Youxia Sun. 2015. The influence of topic description on Chinese student comprehensive essay writing. Foreign Language Teaching and Research 47 (2): 237–250.
Hollingshead, August B., and George Kingsley Zipf. 1949. Human behavior and the principle of least effort: An introduction to human ecology. Cambridge, MA: Addison-Wesley Press.
Karakoç, Nihal Yetkin. 2016. Non-cognitive causes of imprecision in consecutive interpreting in diplomatic settings in light of functionalism. Procedia—Social and Behavioral Sciences 231: 154–158.
Kurz, Ingrid. 2002. Conference interpreting: Quality in the ears of the user. Meta 46 (2): 394–409.
Landauer, Thomas K., and Susan T. Dumais. 1997. A solution to Plato’s problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review 104 (2): 211–240.
Lee, Jieun. 2008. Rating scales for interpreting performance assessment. Interpreter and Translator Trainer 2 (2): 165–184.
Lesch, Harold, and Karen Grové. 2018. Interpreting services in the Western Cape Legislature. Translation and Interpreting Studies 13 (2): 250–268.
Li, Hao. 2014. Rethinking of interpretation quality assessment method—A case study of the 2014 Cross-Straits Chinese-English Interpreting Contest. **an dai jiao ji 7: 27.
Liang, Junying, Qianxi Lv, and Yiguang Liu. 2019. Quantifying interpreting types: Language sequence mirrors cognitive load minimization in interpreting tasks. Frontiers in Psychology 10. https://doi.org/10.3389/fpsyg.2019.00285.
Lin, Yumeng, Qianxi Lv, and Junying Liang. 2018. Predicting fluency with language proficiency, working memory, and directionality in simultaneous interpreting. Frontiers in Psychology 9. https://doi.org/10.3389/fpsyg.2018.01543.
Lu, **nchao. 2018. Propositional information loss in English-to-Chinese simultaneous conference interpreting. Babel 64 (5–6): 792–818.
Lv, Qianxi, and Junying Liang. 2019. Is consecutive interpreting easier than simultaneous interpreting? A corpus-based study of lexical simplification in interpretation. Perspectives 27 (1): 91–106.
McCarthy, Philip M., Gwyneth A. Lewis, David F. Dufty, and Danielle S. McNamara. 2006. Analyzing writing styles with Coh-Metrix. In FLAIRS 2006—Proceedings of the nineteenth international Florida artificial intelligence research society conference. https://www.aaai.org/Papers/FLAIRS/2006/Flairs06-151.pdf.
McNamara, Danielle S., Scott A. Crossley, and Philip M. McCarthy. 2010. Linguistic features of writing quality. Written Communication 27 (1): 57–86.
Meara, Paul. 2005. Designing vocabulary tests for English, Spanish and other languages. In The Dynamics of language use: Functional and contrastive perspectives, ed. Christopher S. Butler, María de los Ángeles Gómez González, and Susana M. Doval-Suárez, 271–285. Amsterdam: John Benjamins.
Ng, Bee Chin. 1992. End users’ subjective reaction to the performance of student interpreters. The Interpreters’ Newsletter Special issue 1: 35–41.
Ouyang, Qianhua. 2018. Assessing meaning-dimension quality in consecutive interpreting training. Perspectives 26 (2): 196–213.
Peng, Gracie. 2009. Using Rhetorical Structure Theory (RST) to describe the development of coherence in interpreting trainees. Interpreting 11 (2): 216–243.
Pöchhacker, Franz. 1994. Quality assurance in simultaneous interpreting. In Teaching translation and interpreting 2: Insights, aims and visions, ed. Cay Dollerup and Annette Lindegaard, 233–244. Amsterdam: John Benjamins.
Saito, Kazuya, Stuart Webb, Pavel Trofimovich, and Talia Isaacs. 2016. Lexical profiles of comprehensible second language speech: The role of appropriateness, fluency, variation, sophistication, abstractness and sense relations. Studies in Second Language Acquisition 38 (4): 677–701.
Shen, Mingxia, and Junying Liang. 2019. Self-repair in consecutive interpreting: Similarities and differences between professional interpreters and student interpreters. Perspectives. https://doi.org/10.1080/0907676X.2019.1701052.
Shen, Mingxia, Qianxi Lv, and Junying Liang. 2019. A corpus-driven analysis of uncertainty and uncertainty management in Chinese premier press conference interpreting. Translation and Interpreting Studies 14 (1): 135–158.
Shlesinger, Miriam. 1994. Intonation in the production and perception of simultaneous interpretation. In Bridging the gap: Empirical research in simultaneous interpretation, ed. Sylvie Lambert and Barbara Moser-Mercer, 225–236. Amsterdam: John Benjamins.
Su, Wei. 2019. Interpreting quality as evaluated by peer students. Interpreter and Translator Trainer 13 (2): 177–189.
Wang, Binhua. 2011. Interpretation quality assessment method and test—A case study of ACIC. Foreign Language World 1: 66–71.
Zhang, Wei. 2008. Client-oriented interpretation quality assessment—A report based on evidence from an interpretation activity. Journal of PLA University of Foreign Languages 31 (5): 84–89.
Zhang, Wei. 2010. Interpretation technology quality assessment: From the perspective of clients. Shanghai Journal of Translators 3: 43–47.
Zhang, Wei. 2011. Conference interpretation quality assessment survey—Comparison between interpreters and clients. Journal of PLA University of Foreign Languages 34 (2): 74–79.
Zhang, Weiwei, and Zhongwei Song. 2019. The effect of self-repair on judged quality of consecutive interpreting: Attending to content, form and delivery. International Journal of Interpreter Education 11 (1): 4–19.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Appendix 9.1 Detailed Description of Coh-Metrix Measures
Appendix 9.1 Detailed Description of Coh-Metrix Measures
The appendix provides a detailed description of the Coh-Metrix indices/variables that appeared as acronyms in the results Sect. 5.1. The Coh-Metrix analysis for the three levels. The information was retrieved from Coh-Metrix website tool documentation based on Graesser et al. (2012).
Coh-Metrix dimension/Index | Description |
|---|---|
Surface Code Level | |
Descriptive | |
DESWC | Number of Words |
Word information | |
WRDFRQc | CELEX Word frequency for content words |
WRDPRO | Pronoun incidence |
WRDPRP1s | First person singular pronoun incidence |
WRDPRP2 | Second person pronoun incidence |
Syntactic pattern density | |
DRINF | Infinitive density incidence |
DRPVAL | Agentless passive voice density incidence |
Syntactic complexity | |
SYNNP | Number of modifiers per noun phrase, mean |
Textbase Level | |
connectives | |
CNCADC | Adversative and contrastive connectives incidence |
CNCAdd | Additive connectives incidence |
CNCCaus | Causal connectives incidence |
CNCLogic | Logical connectives incidence |
CNCTempx | Expanded temporal connectives incidence |
Situation Model Level | |
Text easability principal component scores | |
PCNARp | Text Easability PC Narrativity, percentile |
PCVERBp | Text Easability PC Verb cohesion, percentile |
Readability | |
RDFRE | Flesch Reading Ease |
RDL2 | Coh-Metrix L2 Readability |
Rights and permissions
Copyright information
© 2021 Springer Nature Singapore Pte Ltd.
About this chapter
Cite this chapter
Ouyang, L., Lv, Q., Liang, J. (2021). Coh-Metrix Model-Based Automatic Assessment of Interpreting Quality. In: Chen, J., Han, C. (eds) Testing and Assessment of Interpreting. New Frontiers in Translation Studies. Springer, Singapore. https://doi.org/10.1007/978-981-15-8554-8_9
Download citation
DOI: https://doi.org/10.1007/978-981-15-8554-8_9
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-15-8553-1
Online ISBN: 978-981-15-8554-8
eBook Packages: EducationEducation (R0)