Abstract
One main aim of ILSAs (International Large Scale Assessments) is to develop an empirically based foundation of knowledge for improvement of educational policy and practice. Ideally such knowledge is expressed in causal terms, with statements on what consequences will follow if certain changes are introduced in the educational system. However, such ambitions often seem unrealistic when analyzing ILSA data, because the cross-sectional design of these studies typically only allows investigation of correlations among variables, and correlation must not be confused with causation.
In the first part of the chapter it is clarified why the distinction between correlation and causation is essential, and the reasons why it is generally impossible to answer causal questions through analyses of associations among observed variables are made explicit. Examples are also given of consequences of disregarding the correlational nature of ILSA data. However, within different disciplinary fields alternative techniques have been developed, which under certain assumptions allow causal inferences to be made from nonexperimental data.
Some examples of such techniques are instrumental variable regression, regression discontinuity design, regression with fixed effects, and propensity scoring matching. In the second part of the chapter the basic ideas of some of these analytical approaches are presented and their use is illustrated with data from different ILSAs.
In the third part of the chapter possibilities and limits of causal inference based on ILSA data are discussed.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Allison, P. D. (2009). Fixed effects regression models (Vol. 160). SAGE.
Beaton, A. E., Mullis, I. V., Martin, M. O., Gonzalez, E. J., Kelly, D. L., & Smith, T. A. (1996). Mathematics achievement in the middle school years. IEA’s Third International Mathematics and Science Study (TIMSS). Boston College, Center for the Study of Testing, Evaluation, and Educational Policy.
Biggs, J. (1994). Asian learners through Western eyes: An astigmatic paradox. Australian and New Zealand Journal of Vocational Education Research, 2(2), 40.
Braun, H. I., & Singer, J. D. (2019). Assessment for monitoring of education systems: International comparisons. The Annals of the American Academy of Political and Social Science, 683(1), 75–92.
Brown, T. A. (2015). Confirmatory factor analysis for applied research. Guilford Publications.
Burstein, L. (1980). The analysis of multilevel data in educational research and evaluation, chapter 4. Review of Research in Education, 8(1), 158–233.
Burstein, L. (Ed.). (1992). The IEA Study of Mathematics III. Student growth and classroom processes. Pergamon Press.
Chmielewski, A. K., & Dhuey, K. (2017). The analysis of international large-scale assessments to address causal questions in education policy. Paper commissioned by the National Academy of Education.
Cliffordson, C. (2010). Methodological issues in investigations of the relative effects of schooling and age on school performance: The between-grade regression discontinuity design applied to Swedish TIMSS 1995 data. Educational Research and Evaluation, 16(1), 39–52.
Creemers, B. P., Kyriakides, L., & Sammons, P. (2010). Methodological advances in educational effectiveness research. Routledge.
Duncan, G. J., Magnuson, K. A., & Ludwig, J. (2004). The endogeneity problem in developmental studies. Research in Human Development, 1(1–2), 59–80.
Gustafsson, J.-E. (2010). Longitudinal designs. In B. P. Creemers, L. Kyriakides, & P. Sammons (Eds.), Methodological advances in educational effectiveness research (pp. 77–101). Routledge.
Gustafsson, J. E. (2013). Causal inference in educational effectiveness research: A comparison of three methods to investigate effects of homework on student achievement. School Effectiveness and School Improvement, 24(3), 275–295.
Gustafsson, J.-E., & Nilsen, T. (2016). The impact of school climate and teacher quality on mathematics achievement: A difference-in-differences approach. In T. Nilsen & J.-E. Gustafsson (Eds.), Teacher quality, instructional quality and student outcome. Relationships across countries, cohorts and time (pp. 81–95). Springer International Publishing. https://doi.org/10.1007/978-3-319-41252-8_4
Holland, P. W. (1986). Statistics and causal inference. Journal of the American Statistical Association, 81(396), 945–960.
Kaplan, D., & Lee, C. (2018). Optimizing prediction using Bayesian model averaging: Examples using large-scale educational assessments. Evaluation Review. Advance online publication. https://doi.org/10.1177/0193841X18761421
Klieme, E. (2019). Policies and Practices of Assessment – A showcase for the use (and misuse) of International Large Scale Assessments in Educational Effectiveness Research. In J. Hall, P. Sammons, & A. Lindorff (Eds.), International perspectives in educational effectiveness research. Springer.
Kline, R. B. (2015). Principles and practice of structural equation modeling. Guilford Publications.
Kriegbaum, K., Jansen, M., & Spinath, B. (2015). Motivation: A predictor of PISA’s mathematical competence beyond intelligence and prior test achievement. Learning and Individual Differences, 43, 140–148.
Lazear, E. P. (2001). Educational production. Quarterly Journal of Economics, 116(3), 777–803.
Marsh, H. W., Lüdtke, O., Robitzsch, A., Trautwein, U., Asparouhov, T., Muthén, B., & Nagengast, B. (2009). Doubly-latent models of school contextual effects: Integrating multilevel and structural equation approaches to control measurement and sampling error. Multivariate Behavioral Research, 44(6), 764–802.
Moffitt, R. (2005). Remarks on the analysis of causal relationships in population research. Demography, 42, 91–108. https://doi.org/10.1353/dem.2005.0006
Mosteller, F. (1995). The Tennessee study of class size in the early school grades. The Future of Children, 5(2), 113–127.
Mullis, I. V., Martin, M. O., Foy, P., & Arora, A. (2012). TIMSS 2011 international results in mathematics. TIMSS & PIRLS International Study Center.
Mullis, I. V., Martin, M. O., & Gonzalez, E. J. (2003). PIRLS 2001 international report. International Study Center.
Murnane, R. J., & Willett, J. B. (2010). Methods matter: Improving causal inference in educational and social science research. Oxford University Press.
Pearl, J. (2009). Causality. Cambridge University Press.
Ponzo, M. (2013). Does bullying reduce educational achievement? An evaluation using matching estimators. Journal of Policy Modeling, 35(6), 1057–1078.
Rosén, M., & Gustafsson, J. E. (2016). Is computer availability at home causally related to reading achievement in grade 4? A longitudinal difference in differences approach to IEA data from 1991 to 2006. Large-scale Assessments in Education, 4(1), 5.
Rubin, D. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5), 688–701.
Rubin, D. B. (2006). Matched sampling for causal effects. Cambridge University Press.
Rutkowski, D., & Delandshere, G. (2016). Causal inferences with large scale assessment data: Using a validity framework. Large-scale Assessments in Education, 4, 6.
Seaton, M., Parker, P., Marsh, H. W., Craven, R. G., & Yeung, A. S. (2014). The reciprocal relations between self-concept, motivation and achievement: Juxtaposing academic self-concept and achievement goal orientations for mathematics success. Educational Psychology, 34(1), 49–72.
Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin Company.
Singer, J. D., Braun, H. I., & Chudowsky, N. (2018). International education assessments: Cautions, conundrums, and common sense. National Academy of Education.
Stone, S. I., & Rose, R. A. (2011). Social work research and endogeneity bias. Journal of the Society for Social Work and Research, 2(2), 54–75.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Section Editor information
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this entry
Cite this entry
Gustafsson, JE., Nilsen, T. (2022). Methods of Causal Analysis with ILSA Data. In: Nilsen, T., Stancel-Piątak, A., Gustafsson, JE. (eds) International Handbook of Comparative Large-Scale Studies in Education. Springer International Handbooks of Education. Springer, Cham. https://doi.org/10.1007/978-3-030-38298-8_56-1
Download citation
DOI: https://doi.org/10.1007/978-3-030-38298-8_56-1
Received:
Accepted:
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-38298-8
Online ISBN: 978-3-030-38298-8
eBook Packages: Springer Reference EducationReference Module Humanities and Social SciencesReference Module Education