SpringerLink
Log in

Examining the Effectiveness of a Multicomponent Reading Comprehension Intervention in Middle Schools: A Focus on Treatment Fidelity

  • Research into Practice
  • Published:
Educational Psychology Review Aims and scope Submit manuscript

Abstract

In this experimental study, we examined the effects of a multicomponent reading comprehension intervention in sixth- to eighth-grade English language arts classes with a focus on factors to enhance treatment implementation. We tested the contribution of a theoretically derived fidelity framework that included adherence, quality, dosage, program differentiation, and student responsiveness. The study was conducted in three schools, involving 14 teachers and 859 students. English language arts classes were randomly assigned to the Comprehension Circuit Training intervention (n = 30) or typical practice comparison (n = 31) conditions, and all teachers taught in both conditions. Findings indicated that there was no reliable difference between the intervention and typical practice groups on standardized or researcher-developed proximal measures. To investigate the relationship between fidelity and student outcomes, we conducted a confirmatory factor analysis (CFA) to determine the relation of the indicators to a latent fidelity factor. Results of the CFA revealed a single fidelity factor composed of the adherence, quality, dosage, student responsiveness, and program differentiation variables. The fidelity factor was statistically significantly related to outcomes on a standardized comprehension measure (γ = .86, p < .01) and a narrative measure (γ = .52, p < .01). Results underscore the need to consider the complexity of implementing multicomponent interventions in middle schools and the importance of measuring multiple dimensions of the implementation of fidelity.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Canada)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  • Allinder, R. M., Bolling, R. M., Oats, R. G., & Gagnon, W. A. (2000). Effects of teacher self-monitoring on implementation of curriculum-based measurement and mathematics computation achievement of students with disabilities. Remedial and Special Education, 21, 219–226. doi:10.1177/074193250002100403.

    Article  Google Scholar 

  • Altman, D. G. (1991). Practical statistics for medical research. London: Chapman and Hall.

    Google Scholar 

  • Barnes, M. A., Raghubar, K. P., Faulkner, H., & Denton, C. A. (2014). The construction of visual-spatial situation models in children’s reading and their relation to reading comprehension. Journal of Experimental Child Psychology, 119, 101–111.

    Article  Google Scholar 

  • Beck, I. L., McKeown, M. G., & Kucan, L. (2002). Bringing words to life: Robust vocabulary instruction. New York: Guilford.

  • Benner, G. J., Nelson, J. R., Stage, S. A., & Ralston, N. C. (2011). The influence of fidelity of implementation on the reading outcomes of middle school students experiencing reading difficulties. Remedial and Special Education, 32, 79–88. doi:10.1177/0741932510361265.

    Article  Google Scholar 

  • Browder, D. M., Trela, K., & Jimenez, B. (2007). Training teachers to follow a task analysis to engage middle school students with moderate and severe developmental disabilities in grade-appropriate literature. Focus on Autism and Other Developmental Disabilities, 22(4), 206–219.

    Article  Google Scholar 

  • Brozo, W. G., & Afflerbach, P. P. (2011). Adolescent literacy inventory: grades 6–12. Boston: Pearson.

    Google Scholar 

  • Cantrell, S. C., Almasi, J. F., Carter, J. C., & Rintamaa, M. (2010). Striving readers year 3 evaluation report (Danville, Kentucky). Washington, D.C.: Research report prepared for the U.S. Department of Education.

    Google Scholar 

  • Carlisle, J. F. (2000). Awareness of the structure and meaning of morphologically complex words: impact on reading. Reading and Writing, 12, 169–190.

    Article  Google Scholar 

  • Cash, T. F. (2009). Caveats in the proficient preparation of an APA-style research manuscript for publication. Body Image, 6(1), 1–6. doi:10.1016/j.bodyim.2008.10.003.

    Article  Google Scholar 

  • Catts, H. W., Compton, D., Tomblin, J. B., & Bridges, M. S. (2011). Prevalence and nature of late-emerging poor readers. Journal of Educational Psychology, 104, 166–181. doi:10.1037/a0025323.

    Article  Google Scholar 

  • Chi, M. T. H. (2009). Active-constructive-interactive: a conceptual framework for differentiating learning activities. Topics in Cognitive Science, 1, 73–105. doi:10.1111/j.1756-8765.2008.01005.x.

    Article  Google Scholar 

  • Cromley, J. G., & Azevedo, R. (2007). Testing and refining the direct and inferential mediation model of reading comprehension. Journal of Educational Psychology, 99, 311–325. doi:10.1037/0022-0663.99.2.311.

    Article  Google Scholar 

  • Dane, A. V., & Schneider, B. H. (1998). Program integrity in primary and early secondary prevention: are implementation effects out of control? Clinical Psychology Review, 18(1), 23–45. doi:10.1037/0022-0663.99.2.311.

    Article  Google Scholar 

  • Duke, N. K., Pearson, P. D., Strachan, S. L., & Billman, A. K. (2011). Essential elements of fostering reading comprehension. In S. J. Samuels & A. E. Farstrup (Eds.), What research has to say about reading instruction (4th ed., pp. 51–93). Newark: International Reading Association. Retrieved from http://www.scienceandliteracy.org/sites/scienceandliteracy.org/files/biblio/pdpearson/03-DukePearsonStrachanBillman.prepub.Comprehension.WRST.032311.pdf.

    Chapter  Google Scholar 

  • Edmonds, M. S., Vaughn, S., Wexler, J., Reutebuch, C., Cable, A., Tackett, K. K., et al. (2009). A synthesis of reading interventions and effects on reading comprehenson outcomes for older struggling readers. Review of Educational Research, 79, 262–300. doi:10.3102/0034654308325998.

    Article  Google Scholar 

  • Elleman, A. M., Lindo, E. J., Morphy, P., & Compton, D. L. (2009). The impact of vocabulary instruction on passage-level comprehension of school-age children: a meta-analysis. Journal of Research on Educational Effectiveness, 2, 1–44. doi:10.1080/19345740802539200.

    Article  Google Scholar 

  • Engelmann, S., Hanner, S., & Johnson, G. (2002). Corrective reading. Columbus: SRA/McGraw-Hill.

    Google Scholar 

  • Fixsen, D., Blasé, K., Metz, A., & Van Dyke, M. (2013). Statewide implementation of evidence-based programs. Exceptional Children, 79, 213–230.

    Google Scholar 

  • Garet, M. S., Porter, A. C., Desimone, L., Birman, B. F., & Yoon, K. S. (2001). What makes professional development effective? Results from a national sample of teachers. American Educational Research Journal, 38, 915–945. doi:10.3102/00028312038004915.

    Article  Google Scholar 

  • Garet, M. S., Wayne, A. J., Stancavage, F., Taylor, J., Walters, K., Song, M., et al. (2010). Middle school mathematics professional development impact study: Findings after the first year of implementation. NCEE 2010-4009. National Center for Education Evaluation and Regional Assistance.

  • Gersten, R., Fuchs, L. S., Compton, D., Coyne, M., Greenwood, C., & Innocenti, M. S. (2005). Quality indicators for group experimental and quasi-experimental research in special education. Exceptional Children, 71, 149–164.

    Article  Google Scholar 

  • Graves, M. F., & Prenn, M. C. (1986). Costs and benefits of various methods of teaching vocabulary. Journal of Reading, 29, 596–602.

    Google Scholar 

  • Gresham, F. M. (2009). Evolution of the treatment integrity concept: current status and future directions. School Psychology Review, 38, 533–540.

    Google Scholar 

  • Gresham, F. M., MacMillan, D. L., Beebe-Frankenberger, M. E., & Bocian, K. M. (2000). Treatment integrity in learning disabilities intervention research: do we really know how treatments are implemented? Learning Disabilities Research and Practice, 15(4), 198–205. doi:10.1207/SLDRP1504_4.

    Article  Google Scholar 

  • Guthrie, J. T., McRae, A., & Klauda, S. L. (2007). Contributions of concept-oriented reading instruction to knowledge about interventions for motivations in reading. Educational Psychologist, 42, 237–250. doi:10.1080/00461520701621087.

    Article  Google Scholar 

  • Hulleman, C. S., & Cordray, D. S. (2009). Moving from the lab to the field: the role of fidelity and achieved relative intervention strength. Journal of Research on Educational Effectiveness, 2, 88–110. doi:10.1080/19345740802539325.

    Article  Google Scholar 

  • Institute of Education Sciences (2014). Request for applications: Educational research grants (CFDA Number: 84.305A). Retrieved from http://ies.ed.gov/funding/pdf/2014_84305A.pdf. Accesssed 15 Jan 2014.

  • Jones, H. A., Clarke, A. T., & Power, T. J. (2008). Expanding the concept of intervention integrity: a multidimensional model of participant engagement. Balance, 23, 4–5.

    Google Scholar 

  • Kamil, M. L., Borman, G. D., Dole, J., Kral, C. C., Salinger, T., Torgesen, J., et al. (2008). Improving adolescent literacy: Effective classroom and intervention practices: A Practice Guide (NCEE #2008-4027). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. Retrieved from http://ies.ed.gov/ncee/wwc. Accessed 1 Aug 2013.

  • Kim, A.-H., Vaughn, S., Wanzek, J., & Wei, S. (2004). Graphic organizers and their effects on the reading comprehension of students with LD: a synthesis of research. Journal of Learning Disabilities, 37, 105–118. doi:10.1177/00222194040370020201.

    Article  Google Scholar 

  • Kim, W., Linan–Thompson, S., & Misquitta, R. (2012). Critical factors in reading comprehension instruction for students with learning disabilities: a research synthesis. Learning Disabilities Research and Practice, 27, 66–78. doi:10.1111/j.1540-5826.2012.00352.x.

    Article  Google Scholar 

  • King, A. (1994). Guiding knowledge construction in the classroom: effects of teaching children how to question and how to explain. American Educational Research Journal, 31, 338–368. doi:10.2307/1163313.

    Article  Google Scholar 

  • King, A. (2002). Structuring peer interaction to promote high-level cognitive processing. Theory Into Practice, 41, 33–39. doi:10.1207/s15430421tip4101_6.

    Article  Google Scholar 

  • Kintsch, W. (1998). The role of knowledge in discourse comprehension: a construction-integration model. Psychological Review, 95, 163–183. doi:10.1037/0033-295X.95.2.163.

    Article  Google Scholar 

  • Kline, P. (2000). The handbook of psychological testing (2nd ed.). London: Routledge.

    Google Scholar 

  • Klingner, J. K., Vaughn, S., Arguelles, M. E., Hughes, M. T., & Leftwich, S. A. (2004). Collaborative strategic reading: “real-world” lessons from classroom teachers. Remedial and Special Education, 25, 291–302. doi:10.1177/07419325040250050301.

    Article  Google Scholar 

  • Kushman, J., Hanita, M., & Raphael, J. (2011). An experimental study of the project CRISS reading program on grade 9 reading achievement in rural high schools. (NCEE 2011-4007). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

  • Lipsey, M. W., Puzio, K., Yun, C., Hebert, M. A., Steinka-Fry, K., Cole, M. W., et al. (2012). Translating the statistical representation of the effects of education interventions into more readily interpretable forms. Washington, DC: National Center for Special Education Research.

  • MacGinitie, W. H., MacGinitie, R. K., Maria, K., & Dreyer, L. G. (2010). Gates-MacGinitie Reading Tests (4th ed.). Itasca: Riverside Publishing.

    Google Scholar 

  • McCormick, S. (1989). Effects of previews on more skilled and less skilled readers’ comprehension of expository text. Journal of Literacy Research, 21, 219–239. doi:10.1080/10862968909547674.

    Article  Google Scholar 

  • McMaster, K. L., van den Broek, P. V., Espin, C. A., White, M. J., Rapp, D. N., Kendeou, P., et al. (2012). Making the right connections: Differential effects of reading intervention for subgroups of comprehenders. Learning and Individual Differences, 22, 100–111.

  • National Center for Education Statistics. (2013). The nation’s report card: a first look: 2013 mathematics and reading (NCES 2014-451). Washington, DC: U.S. Department of Education, Institute of Education Sciences.

    Google Scholar 

  • National Governors Association Center for Best Practices, Council of Chief State School Officers (2010). Common core state standards for English Language arts and literacy in history/social studies, science, and technical subjects: Appendix B: Text exemplars and sample performance tasks. Washington, DC: National Governors Association Center for Best Practices, Council of Chief State School Officers. Retrieved from http://www.corestandards.org/assets/Appendix_B.pdf. Accessed 25 July 2013.

  • Nunnally, J. C. (1978). Psychometric theory (2nd ed.). New York: McGraw Hill.

    Google Scholar 

  • O’Donnell, C. L. (2008). Defining, conceptualizing, and measuring fidelity of implementation and its relationship to outcomes in K-12 curriculum intervention research. Review of Educational Research, 78, 33–84. doi:10.3102/0034654307313793.

    Article  Google Scholar 

  • Oakhill, J. V., & Cain, K. (2012). The precursors of reading ability in young readers: evidence from a four-year longitudinal study. Scientific Studies of Reading, 16, 91–121. doi:10.1080/10888438.2010.529219.

    Article  Google Scholar 

  • Penuel, W. R., & Means, B. (2004). Implementation variation and fidelity in an inquiry science program: analysis of GLOBE data reporting patterns. Journal of Research in Science Teaching, 41, 294–315. doi:10.1002/tea.20002.

    Article  Google Scholar 

  • Penuel, W. R., Fishman, B. J., Yamaguchi, R., & Gallagher, L. P. (2007). What makes professional development effective? Strategies that foster curriculum implementation. American Educational Research Journal, 44, 921–958. doi:10.3102/0002831207308221.

    Article  Google Scholar 

  • Penuel, W. R., Riel, M., Krause, A. E., & Frank, K. A. (2009). Analyzing teachers’ professional interactions in a school as social capital: a social network approach. The Teachers College Record, 111, 124–163.

    Google Scholar 

  • Rapp, D. N., van den Broek, P., McMaster, K. L., Kendeou, P., & Espin, C. A. (2007). Higher-order comprehension processes in struggling readers: a perspective for research and intervention. Scientific Studies of Reading, 11, 289–312. doi:10.1080/10888430701530417.

    Article  Google Scholar 

  • Robinson, J. P., Shaver, P. R., & Wrightsman, L. S. (Eds.). (1991). Measures of personality and social psychological attitudes. San Diego: Academic.

    Google Scholar 

  • Roscoe, R. D., & Chi, M. T. H. (2007). Understanding tutor learning: knowledge-building and knowledge-telling in peer tutors’ explanation and questions. Review of Educational Research, 77, 534–574. doi:10.3102/0034654307309920.

    Article  Google Scholar 

  • Sailors, M., & Price, L. R. (2010). Professional development that supports the teaching of cognitive reading strategy instruction. The Elementary School Journal, 110, 301–322. doi:10.1086/648980.

    Article  Google Scholar 

  • Scammacca, N., Roberts, G., Vaughn, S., Edmonds, M., Wexler, J., Reutebuch, C. K., et al. (2007). Interventions for adolescent struggling readers: a meta-analysis with implications for practice. Portsmouth, NH: RMC Research Corporation, Center on Instruction.

    Google Scholar 

  • Scher, L., & O'Reilly, F. (2009). Professional development for K-12 math and science teachers: what do we really know? Journal of Research on Educational Effectiveness, 2, 209–249. doi:10.1080/19345740802641527.

    Article  Google Scholar 

  • Schulte, A. C., Easton, J. E., & Parker, J. (2009). Advances in treatment integrity research: multidisciplinary perspectives on the conceptualization, measurement, and enhancement of treatment integrity. School Psychology Review, 38, 460–475.

    Google Scholar 

  • Sheridan, S. M., Swanger-Gagne, M., Welch, G. W., Kwon, K., & Garbacz, S. A. (2009). Fidelity measurement in consultation: psychometric issues and preliminary examination. School Psychology Review, 38, 476–495.

    Google Scholar 

  • Simmons, D., Fogarty, M., Oslund, E., Simmons, L., Hairrell, A., Davis, J., et al. (2014). Integrating content knowledge-building and student-regulated comprehension practices in secondary English language arts classes. Journal of Research on Educational Effectiveness. doi:10.1080/19345747.2013.836766.

    Google Scholar 

  • Stahl, S. A., & Fairbanks, M. M. (1986). The effects of vocabulary instruction: a model-based meta-analysis. Review of Educational Research, 56, 72–110. doi:10.3102/00346543056001072.

    Article  Google Scholar 

  • Supovitz, J. A., & Turner, H. M. (2000). The effects of professional development on science teaching practices and classroom culture. Journal of Research in Science Teaching, 37, 963–980. doi:10.1002/1098-2736(200011)37:9<963::AID-TEA6>3.0.CO;2-0.

    Article  Google Scholar 

  • Swanson, E., Wanzek, J., Haring, C., Ciullo, S., & McCulley, L. (2013). Intervention fidelity in special and general education research journals. The Journal of Special Education, 47, 3–13. doi:10.1177/0022466911419516.

    Article  Google Scholar 

  • Taboada, A., Townsend, D., & Boynton, M. J. (2013). Mediating effects of reading engagement on the reading comprehension of early adolescent English language learners. Reading and Writing Quarterly, 29, 309–332. doi:10.1080/10573569.2013.741959.

    Article  Google Scholar 

  • Torgesen, J. K. (1977). The role of nonspecific factors in the task performance of learning disabled children: a theoretical assessment. Journal of Learning Disabilities, 10, 27–34. doi:10.1177/002221947701000108.

    Article  Google Scholar 

  • van den Broek, P., Rapp, D. N., & Kendeou, P. (2005). Integrating memory-based and constructionist processes in accounts of reading comprehension. Discourse Processes, 39, 299–316. doi:10.1080/0163853X.2005.9651685.

    Article  Google Scholar 

  • Vaughn, S., Klingner, J. K., Swanson, E. A., Boardman, A. G., Roberts, G., Mohammed, S. S., et al. (2011). Efficacy of collaborative strategic reading with middle school students. American Educational Research Journal, 48, 938–964. doi:10.3102/0002831211410305.

    Article  Google Scholar 

  • Vaughn, S., Roberts, G., Klingner, J. K., Swanson, E. A., Boardman, A., Stillman-Spisak, S. J., et al. (2013). Collaborative strategic reading: Findings from experienced implementers. Journal of Research on Educational Effectiveness, 6, 137–163. doi:10.1080/19345747.2012.741661.

  • Ysseldyke, J., Spicuzza, R., Kosciolek, S., Teelucksingh, E., Boys, C., & Lemkuil, A. (2003). Using a curriculum-based instructional management system to enhance math achievement in urban schools. Journal of Education for Students Placed at Risk, 8, 247–265. doi:10.1207/S15327671ESPR0802_4.

    Article  Google Scholar 

Download references

Acknowledgments

The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305F100013 to Texas A&M University as part of the Reading for Understanding Research Initiative. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.

Author information

Authors and Affiliations

  1. Department of Educational Psychology, Texas A&M University, 4225 TAMU, College Station, TX, 77843-4225, USA

    Melissa Fogarty, Deborah Simmons, John Davis, Leslie Simmons, Leah Anderson & Nathan Clemens

  2. The Meadows Center for Preventing Educational Risk, The University of Texas at Austin, College of Education SZB 228, 1912 Speedway, D4900, Austin, TX, 78712-1284, USA

    Eric Oslund

  3. Meadows Center and Vaughn Gross Center, College of Education, University of Texas at Austin, 1 University Station D4900, Austin, TX, 78712, USA

    Greg Roberts

Authors
  1. Melissa Fogarty
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Eric Oslund
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Deborah Simmons
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. John Davis
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Leslie Simmons
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Leah Anderson
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Nathan Clemens
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Greg Roberts
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Melissa Fogarty.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fogarty, M., Oslund, E., Simmons, D. et al. Examining the Effectiveness of a Multicomponent Reading Comprehension Intervention in Middle Schools: A Focus on Treatment Fidelity. Educ Psychol Rev 26, 425–449 (2014). https://doi.org/10.1007/s10648-014-9270-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10648-014-9270-6

Keywords

Search

Navigation