Abstract
Although the recent identification of the five critical components of early literacy has been a catalyst for modifications to the content of materials used to provide reading instruction and the tools used to examine student’s acquisition of early literacy skills, these skills have not received equal attention from test developers and publishers. In particular, a review of early literacy available measures for screening and monitoring students reveals a dearth of tools for examining different facets of reading comprehension. The purposes of this study were twofold: (a) to examine the relative difficulty of items written to assess literal, inferential, and evaluative comprehension, and (b) to compare single factor and bifactor models of reading comprehension to determine if items written to assess students’ literal, inferential, and evaluative comprehension abilities comprise unique measurement factors. Data from approximately 2,400 fifth grade students collected in the fall, winter, and spring of fifth grader were used to examine these questions. Findings indicated that (a) the relative difficulty of item types may be curvilinear, with literal items being significantly less challenging than inferential and evaluative items, and (b) literal, inferential, and evaluative comprehension measurement factors explained unique portions of variance in addition to a general reading comprehension factor. Instructional implications of the findings are discussed.
Similar content being viewed by others
References
Adams, M. J. (1990). Beginning to read: Thinking and learning about print. Cambridge, MA: Massachusetts Institute of Technology.
Allington, R. L., Chodos, L., Domaracki, J., & Trueax, S. (1977). Passage dependency: Four diagnostic oral reading tests. The Reading Teacher, 30, 369–375.
Alonzo, J., Basaraba, D., Tindal, G., & Carriveau, R. (2009). They read, but how well do they understand? An empirical look at the nuances of measuring reading comprehension. Assessment for Effective Intervention, 35, 34–44.
Alonzo, J., Liu, K., & Tindal, G. (2007). Examining the technical adequacy of reading comprehension measures in a progress monitoring assessment system. Technical report #41. Eugene, OR: Behavioral Research and Teaching, University of Oregon.
Alonzo, J., & Tindal, G. (2008). Examining the technical adequacy of fifth-grade reading comprehension measures in a progress monitoring assessment system. Technical report no. 0808. Eugene, OR: Behavioral Research and Teaching, University of Oregon.
Alonzo, J., & Tindal, G. (2009). Alternate form and test–retest reliability of easyCBM reading measures. Technical report no. 0906. Eugene, OR: Behavioral Research and Teaching, University of Oregon.
Alonzo, J., Tindal, G., Ulmer, K., & Glasgow, A. (2006). easyCBM online progress monitoring assessment system. Eugene, OR: Center for Educational Assessment Accountability. Available at http://easycbm.com.
Anderson, R. C. (2004). Role of the reader’s schema in comprehension, learning, and memory. In R. B. Ruddell & N. J. Unrau (Eds.), Theoretical models and process of reading (5th ed., pp. 594–619). Newark, DE: International Reading Association.
Anderson, D., Alonzo, J., & Tindal, G. (2011a). easyCBM reading criterion related validity evidence: Oregon state test 2009–2010. Technical report no. 1103. Eugene, OR: Behavioral Research and Teaching, University of Oregon.
Anderson, D., Alonzo, J., & Tindal, G. (2011b). easyCBM reading criterion related validity evidence: Washington state test 2009–2010. Technical report no. 1101. Eugene, OR: Behavioral Research and Teaching, University of Oregon.
Applegate, M. D., Quinn, K. B., & Applegate, A. J. (2002). Levels of thinking required by comprehension questions in informal inventories. The Reading Teacher, 56, 174–180.
Ardoin, S. P., Williams, J. C., Christ, T. J., Klubnik, C., & Wellborn, C. (2010). Examining readability estimates’ predictions of students’ oral reading rate: Spache, Lexile, and Forcast. School Psychology Review, 39, 277–285.
Bailin, A., & Grafstein, A. (2001). The linguistic assumption underlying readability formulae: A critique. Language & Communication, 21, 285–301.
Baker, S. K., Smolkowski, K., Katz, R., Fien, H., Seeley, J. R., Kame’enui, E. J., et al. (2008). Reading fluency as a predictor of reading proficiency in low-performing, high-poverty schools. School Psychology Review, 37, 18–37.
Basaraba, D., Travers, P., & Chaparro, E. (February, 2011). Application of Ehri’s theory: Instructional implications of students’ decoding skills. Paper presented at the National Association of School Psychology annual conference, San Diego, CA.
Best, R. M., Floyd, R. G., & McNamara, D. S. (2008). Differential competencies contributing to children’s comprehension of narrative and expository texts. Reading Psychology, 29, 137–164.
Boulineau, T., Fore, C., Hagan-Burke, S., & Burke, M. D. (2004). Use of story-map** to increase the story-grammar text comprehension of elementary students with learning disabilities. Learning Disability Quarterly, 27, 105–121.
Bozdogan, H. (2000). Akaike’s information criterion and recent developments in information complexity. Journal of Mathematical Psychology, 44, 62–91.
Bruce, B., & Rubin, A. (1988). Readability formulas: Matching tool and task. In A. Davison & G. M. Green (Eds.), Linguistic complexity and text comprehension: Readability issues reconsidered (pp. 5–22). Hillsdale, NJ: Lawrence Erlbaum.
Burkam, D. T., Ready, D. D., Lee, V. E., & LoGerfo, L. F. (2004). Social-class differences in summer learning between kindergarten and first grade: Model specification and estimation. Sociology of Education, 77, 1–31.
Bus, A. G., & van Ijzendoorn, M. H. (1999). Phonological awareness and early reading: A meta-analysis of experimental training studies. Journal of Educational Psychology, 91, 403–414.
Cain, K., Oakhill, J., Barnes, M. A., & Bryant, P. (2001). Comprehension skill, inference-making ability, and their relation to knowledge. Memory & Cognition, 29, 850–859.
Cain, K., Oakhill, J., & Bryant, P. (2004). Children’s reading comprehension ability: Concurrent prediction by working memory, verbal ability, and component skills. Journal of Educational Psychology, 96, 31–42.
Campbell, J. R. (2005). Single instrument, multiple measures: Considering the use of multiple item formats to assess reading comprehension. In S. G. Paris & S. A. Stahl (Eds.), Children’s reading comprehension and assessment (pp. 347–368). Mahwah, NJ: Lawrence Erlbaum.
Carnine, D. W., Silbert, J., Kame’enui, E. J., & Tarver, S. G. (2010). Direct instruction reading (5th ed.). Boston, MA: Merrill.
Chard, D. J., Simmons, D. C., & Kame’enui, E. J. (1998). Word recognition: Research bases. In D. C. Simmons & E. J. Kame’enui (Eds.), What reading research tells us about children with diverse learning needs: Bases and basics (pp. 141–168). Mahwah, NJ: Lawrence Erlbaum.
Chen, F. F., West, S. G., & Sousa, K. H. (2006). A comparison of bifactor and second-order models of quality of life. Multivariate Behavioral Research, 41, 189–225.
Chikalanga, I. (1992). A suggested taxonomy of inferences for the reading teacher. Reading in a Foreign Language, 8, 697–709.
Christ, T. J., & Ardoin, S. P. (2009). Curriculum-based measurement of oral reading: Passage equivalence and probe-set development. Journal of School Psychology, 47, 55–75.
Coleman, C., Lindstrom, J., Nelson, J., Lindstrom, W., & Gregg, K. N. (2010). Passageless comprehension on the Nelson-Denny Reading Test: Well above chance for university students. Journal of Learning Disabilities, 43, 244–249.
Davey, B. (1988). Factors affecting the difficulty of reading comprehension items for successful and unsuccessful readers. Journal of Experimental Education, 56, 67–75.
Deno, S. L. (1985). Curriculum-based measurement: The emerging alternative. Exceptional Children, 52, 219–232.
Dimino, J., Gersten, R., Carnine, D., & Blake, G. (1990). Story grammar: An approach for promoting at-risk secondary students’ comprehension of literature. The Elementary School Journal, 91, 19–32.
Dole, J. A., Duffy, G. G., Roehler, L. R., & Pearson, P. D. (1991). Moving from the old to the new: Research on reading comprehension instruction. Review of Educational Research, 61, 239–264.
Duran, N. D., McCarthy, P. M., Graesser, A. C., & McNamara, D. S. (2007). Using temporal cohesion to predict temporal coherence in narrative and expository texts. Behavior Research Methods, 39, 212–223.
Durkin, D. (1989). Teaching them to read (5th ed.). Boston, MA: Allyn & Bacon.
Fien, H., Baker, S. K., Smolkowski, K., Mercier Smith, J. L., Kame’enui, E. J., & Thomas Beck, C. (2008). Using nonsense word fluency to predict reading proficiency in kindergarten through second grade for English learners and native English speakers. School Psychology Review, 37, 391–408.
Fitzgerald, T. P., & Fitzgerald, E. F. (1978). A cross-cultural study of three measures of comprehension at the primary and intermediate levels. Educational Research Quarterly, 3, 84–89.
Fox, E. (2002). The role of reader characteristics in processing and learning from informational text. Review of Educational Research, 79, 197–261.
Francis, D. J., Santi, K. L., Barr, C., Fletcher, J. M., Varisco, A., & Foorman, B. (2008). Form effects on the estimation of students’ oral reading fluency using DIBELS. Journal of School Psychology, 46, 315–342.
Freedle, R., & Kostin, I. (1999). Does the text matter in a multiple-choice test of comprehension? The case for the construct validity of TOEFL’s mini talks. Language Testing, 16(2), 2–32.
Gernsbacher, M. A., Robertson, R. R., Palladino, P., & Werner, N. K. (2004). Managing mental representations during narrative comprehension. Discourse Processes, 5, 53–72.
Gersten, R., Fuchs, L. S., Williams, J. P., & Baker, S. K. (2001). Teaching reading comprehension strategies to students with learning disabilities: A review of the research. Review of Educational Research, 71, 279–320.
Gibbons, R. D., Bock, R. D., Hedeker, D., Weiss, D. J., Segawa, E., Bhaumik, D. K., et al. (2006). Full information item bifactor analysis of graded response data. Applied Psychological Measurement, 31, 4–19.
Gibbons, R. D., & Hedeker, D. R. (1992). Full-information item bifactor analysis. Psychometrika, 57, 423–436.
Good, R. H., Baker, S. K., & Peyton, J. A. (2009). Making sense of nonsense word fluency: Determining adequate progress in early first-grade reading. Reading and Writing Quarterly, 25, 33–56. doi:10.1080/1058/3560802491224.
Good, R. H., & Kaminski, R. A. (2011). DIBELS next assessment manual. Eugene, OR: Dynamic Measurement Group.
Good, R., Simmons, D., & Kame’enui, E. (2001). The importance and decision-making utility of a continuum of fluency-based indicators of foundational reading skills for third-grade high-stakes outcomes. Scientific Reading Studies, 5, 257–288.
Graesser, A. C., Singer, M., & Trabasso, T. (1994). Constructing inferences during narrative text comprehension. Psychological Review, 101, 371–395.
Hannon, B., & Daneman, M. (2001). A new tool for measuring and understanding individual differences in the component processes of reading comprehension. Journal of Educational Psychology, 93, 103–128.
Harn, B. A., Stoolmiller, M., & Chard, D. J. (2008). Measuring the dimensions of alphabetic principle on the reading development of first graders: The role of automaticity and unitization. Journal of Learning Disabilities, 41, 143–157.
Herber, H. L. (1970). Teaching reading in the content areas. Englewood Cliffs, NJ: Prentice Hall.
Kendall, J. R., Mason, J. M., & Hunter, W. (1980). Which comprehension? Artifacts in the measurement of reading comprehension. The Journal of Educational Research, 73, 233–236.
Kintsch, W. (1988). The role of knowledge in discourse comprehension: A construction-integration model. Psychological Review, 95, 163–182.
Kintsch, W., & Kintsch, E. (2005). Comprehension. In S. G. Paris & S. A. Stahl (Eds.), Children’s reading comprehension and assessment (pp. 71–92). New York, NY: Routledge.
Kintsch, W., & Rawson, K. A. (2005). Comprehension. In M. J. Snowling & C. Hulme (Eds.), The science of reading: A handbook (pp. 209–226). Malden, MA: Blackwell.
LaBerge, D., & Samuels, S. J. (1974). Toward a theory of automatic information processing in reading. Cognitive Psychology, 6, 293–323.
Lapp, D., & Flood, J. (1983). Teaching reading to every child. New York: MacMillan.
Lawrence, J. F. (2011). English vocabulary trajectories of students whose parents speak a language other than English: Steep trajectories and sharp summer setback. Reading and Writing: An Interdisciplinary Journal. doi:10.1007/s11145-011-9305-z.
Leu, D. J., & Kinzer, C. K. (1999). Effective literacy instruction (K-8) (4th ed.). Upper Saddle River, NJ: Prentice Hall.
Masson, M. E., & Miller, J. A. (1983). Working memory and individual differences in comprehension and memory of text. Journal of Educational Psychology, 75, 314–318.
McCoach, D. B., O’Connell, A. A., Reis, S. M., & Levitt, H. A. (2006). Growing readers: A hierarchical linear model on children’s reading growth during the first 2 years of school. Journal of Educational Psychology, 98, 14–28.
McCormick, S. (1992). Disabled readers’ erroneous responses to inferential comprehension questions: Description and analyses. Reading Research Quarterly, 27(1), 54–77.
McNamara, D. S. (1997). Comprehension skill: A knowledge-based account. In Proceedings of the 19th Annual Meeting of the Cognitive Science Society (pp. 508–513). Hillsdale, NJ: Erlbaum.
McNamara, D. S., & Magliano, J. (2009). Toward a comprehensive model of reading comprehension. In B. Ross (Ed.), The psychology of learning and motivation (Vol. 51, pp. 297–384). New York, NY: Elsevier Science.
Muthén, L. K., & Muthén, B. O. (2010). Mplus: Statistical Analysis with latent variables (version 5.21). Los Angeles, CA: Muthén & Muthén.
Nation, K. (2005). Children’s reading comprehension difficulties. In M. J. Snowling & C. Hulme (Eds.), The science of reading: A handbook (pp. 248–265). Oxford, MA: Blackwell.
National Institute of Child Health and Human Development (NICHD). (2000). Report of the National Reading Panel. Teaching children to read: An evidence-based assessment of scientific research literature on reading and its implications for instruction: Reports of the subgroups. NIH publication no. 00-4754. Washington, DC: U.S. Government Printing Office. Also available on-line: http://www.nichd.nih.gov/publications/nrp/report.htm.
National Research Council. (1998). Organizational strategies for kindergarten and the primary grades. In C. E. Snow, M. S. Burns, & P. Griffin (Eds.), Preventing reading difficulties in young children. Washington, DC: National Academy Press.
No Child Left Behind Act of 2001: Reading first, 20 U.S.C. § 6319 (2008).
Oakhill, J., & Cain, K. (2007). Issues of causality in children’s reading comprehension. In D. S. McNamara (Ed.), Reading comprehension strategies: Theories, interventions, and technologies (pp. 47–72). Mahwah, NJ: Lawrence Erlbaum.
Oakhill, J., Hartt, J., & Samols, D. (2005). Levels of comprehension monitoring and working memory in good and poor comprehenders. Reading and Writing, 18, 657–686.
Otero, J., Léon, J. A., & Graesser, A. C. (Eds.). (2002). The psychology of science text comprehension. Mahwah, NJ: Lawrence Erlbaum.
Pearson, P. D., & Fielding, L. (1991). Comprehension instruction. In R. Barr, M. L. Kamil, P. Mosenthal, & P. D. Pearson (Eds.), Handbook of reading research (Vol. 2, pp. 815–860). White Plains, NY: Longman.
Pearson, P. D., & Hamm, D. N. (2005). The assessment of reading comprehension: A review of practices—Past, present, and future. In S. G. Paris & S. A. Stahl (Eds.), Children’s reading comprehension and assessment (pp. 13–70). Mahwah, NJ: Lawrence Erlbaum.
Pearson, P. D., & Johnson, D. D. (1978). Teaching reading comprehension. New York, NY: Holt, Rinehart, & Winston.
Perfetti, C. A. (1985). Reading ability. New York: Oxford University Press.
Perfetti, C. A. (1999). Comprehending written language: A blueprint of the reader. In C. M. Brown & P. Hagoart (Eds.), The neurocognition of language (pp. 167–208). New York, NY: Oxford University Press.
Perfetti, C. A., Landi, N., & Oakhill, J. (2005). The acquisition of reading comprehension skill. In M. J. Snowling & C. Hulme (Eds.), The science of reading: A handbook (pp. 227–247). Malden, MA: Blackwell.
Rayner, K., Foorman, B. R., Perfetti, C. A., Pesetsky, D., & Seidenberg, M. S. (2001). How psychological science informs the teaching of reading. Psychological Science in the Public Interest, 2, 31–74.
Rupley, W. H., & Blair, T. R. (1983). Reading diagnosis and remediation: Classroom and clinic (2nd ed.). Boston: Houghton Mifflin.
Rupley, W. H., & Willson, V. L. (1996). Content, domain, and word knowledge: Relationship to comprehension of narrative and expository text. Reading and Writing: An Interdisciplinary Journal, 8, 419–432.
Rupp, A. A., Ferne, T., & Choi, H. (2006). How assessing reading comprehension with multiple-choice questions shapes the construct: A cognitive processing perspective. Language Testing, 23(4), 441–474.
Saéz, L., Park, B., Nese, J. F., Jamgochian, E., Lai, C-F., Anderson, D., et al. (2011). Technical adequacy of the easyCBM reading measures (grades 3–7), 2009–2010 version. Technical report #1005. Eugene, OR: Behavioral Research and Teaching, University of Oregon.
Sarroub, L., & Pearson, P. D. (1998). Two steps forward, three steps back: The stormy history of reading comprehension assessment. The Clearinghouse, 72, 97–105.
Shinn, M. R., & Shinn, M. M (2002). AIMSweb training workbook: Administration and scoring of reading MAZE for use in general outcomes measurement. Retrieved August 25, 2011 from http://www.aimsweb.com/uploads/pdfs/scoring_maze.pdf.
Short, E. J., & Ryan, E. B. (1984). Metacognitive differences between skilled and less skilled readers: Remediating deficits through story grammar and attribution training. Journal of Educational Psychology, 76, 225–235.
Simms, L. J., Grös, D. F., Watson, D., & O’Hara, M. W. (2008). Parsing the general and specific components of depression and anxiety with bifactor modeling. Depression and Anxiety, 25, E34–E46.
Smith, S. B., Simmons, D. C., & Kame’enui, E. J. (1998). Phonological awareness: Research bases. In D. C. Simmons & E. J. Kame’enui (Eds.), What reading research tells us about children with diverse learning needs: Bases and basics (pp. 61–128). Mahwah, NJ: Lawrence Erlbaum.
Snider, V. E. (Fall 1988). The role of prior knowledge in reading comprehension: A test with LD adolescents. Direct Instruction News, 6–11.
Snow, C. E. (2002). Reading for understanding: Toward an R&D program in reading comprehension. Santa Monica, CA: RAND.
Stothard, S. E., & Hulme, C. (1996). A comparison of reading comprehension and decoding difficulties in children. In C. Cornoldi & J. Oakhill (Eds.), Reading comprehension difficulties: Processes and intervention (pp. 93–112). Hillsdale, NJ: Lawrence Erlbaum.
Swanson, H. L., & O’Connor, R. E. (2009). The role of working memory and fluency training on reading comprehension in children who are dysfluent readers. Journal of Learning Disabilities, 42, 548–575.
Sweet, A. P. (2005). Assessment of reading comprehension: The RAND reading study group vision. In S. G. Paris & S. A. Stahl (Eds.), Children’s reading comprehension and assessment. Mahwah, NJ: Lawrence Erlbaum.
Tal, N. F., Siegel, L. S., & Maraun, M. (1994). The role of question type and reading ability in reading comprehension. Reading and Writing: An Interdisciplinary Journal, 6, 387–402.
Taylor, B. M., & Beach, R. W. (1984). The effects of text structure on middle-grade students’ comprehension and production of expository text. Reading Research Quarterly, 19, 134–146.
Taylor, S. E., Frackenpohl, H., & White, C. E. (1989). EDL core vocabularies in reading, mathematics, science, and social studies. Orlando, FL: Steck-Vaughn.
Tuinman, J. J. (1973). Determining the passage dependency of comprehension questions in 5 major tests. Reading Research Quarterly, 9, 207–223.
Tun, P. A. (1989). Age differences in processing expository and narrative text. Journal of Gerontology, 44, 9–15.
Vacca, J., Vacca, R. T., & Gove, M. K. (1987). Reading and learning to read. Boston, MA: Little, Brown, & Co.
Vacca, J. L., Vacca, R. T., Gove, M. K., Burkey, L. C., Lenhart, L. A., & McKeon, C. A. (2009). Reading and learning to read (7th ed.). Boston, MA: Pearson.
van den Bergh, H. (1990). On the construct validity of multiple-choice items for reading comprehension. Applied Psychological Measurement, 14, 1–12.
van den Broek, P., Lorch, R. P., Linderholm, T., & Gustafson, M. (2001a). The effects of readers’ goals on inference generation and memory for texts. Memory & Cognition, 29, 1081–1087.
van den Broek, P., Tzeng, Y., Risden, K., Trabasso, T., & Basche, P. (2001b). Inferential questioning: Effects of comprehension of narrative texts as a function of grade and timing. Journal of Educational Psychology, 93, 521–529.
Van Dijk, T. A., & Kintsch, W. (1983). Strategies of discourse comprehension. New York, NY: Academic Press.
Weaver, C. A., & Kintsch, W. (1991). Expository text. In R. Barr, M. L. Kamil, P. Mosenthal, & P. D. Pearson (Eds.), Handbook of reading research (Vol. 2, pp. 230–244). White Plains, NY: Longman.
Wood, R., Wilson, D. T., Gibbons, R., Schilling, S., Muraki, E., & Bock, R. D. (2003). TESTFACT 4 for windows: Test scoring, item statistics, and item factor analysis (computer software). Lincolnwood, IL: Scientific Software International, Inc.
Yovanoff, P., Duesbery, L., Alonzo, J., & Tindal, G. (2005). Grade-level invariance of a theoretical causal structure predicting reading comprehension with vocabulary and oral reading fluency. Educational Measurement: Issues and Practice, 24, 4–12.
Zvoch, K. (2009). A longitudinal examination of the academic year and summer learning rates of full- and half-day kindergarteners. Journal of Education for Students Placed at Risk, 14, 311–333.
Zwaan, R. A., & Brown, C. M. (1996). The influence of language proficiency and comprehension skill on situation-model construction. Discourse Processes, 21, 289–327.
Zwaan, R. A., Radvansky, G. A., Hilliard, A. E., & Curiel, J. M. (1998). Constructing multidimensional situation models during reading. Scientific Studies of Reading, 2, 199–220.
Acknowledgments
This study was partially funded by a grant from the Office of Special Education Programs: Model Demonstration Centers on Progress Monitoring (Award H326M050003). The findings and conclusions, however, are those of the authors only and do not necessarily represent the perspectives of the U.S. Department of Education or the Office of Special Education Programs.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Basaraba, D., Yovanoff, P., Alonzo, J. et al. Examining the structure of reading comprehension: do literal, inferential, and evaluative comprehension truly exist?. Read Writ 26, 349–379 (2013). https://doi.org/10.1007/s11145-012-9372-9
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11145-012-9372-9