Abstract
We specify a three-factor random effects model from a reliability study, where the effects of subjects, raters, and items are random. The reliability measure of interest is an intraclass correlation coefficient that measures the relative agreement of a single measurement on an individual from a randomly selected rater on a randomly selected item. Our objective is to evaluate and illustrate an approximate confidence interval for this intraclass correlation coefficient based on Satterthwaite’s approximation (Wong and McGraw, Educational and Psychological Measurement, 59:270–288, 1999). In doing so, we perform Monte Carlo simulations and provide an illustration. Overall, the actual coverage of one-sided 95 % lower bounds and upper bounds, along with two-sided 90 % confidence intervals, for this particular intraclass correlation coefficient aligns with the nominal coverage for the commonly applied settings evaluated. This methodological evaluation is, to our knowledge, the first to validate the method.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Adamec E, Burdick R (2003) Confidence intervals for a discrimination ratio in a gauge R&R study with three random factors. Qual Eng 15(3):283–389
Armitage P, Berry G (1994) Statistical methods in medical research, 3rd edn. Blackwell, Oxford
Arteaga C, Jeyaratnam S, Graybill FA: (1982) Confidence intervals for proportions of total variance in the two-way cross component of variance model. Comm Stat Theor M 11(15):1643–1658
Brennan RL (2001) Generalizability theory. Springer, New York
Cappelleri JC, Ting N (2003) A modified large-sample approach to approximate interval estimation for a particular intraclass correlation coefficient. Stat Med 22(11):1861–1877
Fleiss JL, Shrout PE (1978) Approximate interval estimation for a certain intraclass correlation coefficient. Psychometrika 43(2):259–262
Fleiss JL (1986) The Design and analysis of clinical experiments. Wiley, New York
Hamada M, Weerahandi S (2000) Measurement system assessment via generalized inference. J Qual Tech 32(3):241–253
Hayes K, Walton JR, Szomor ZL, Murrell GAC (2002) Reliability of 3 methods for assessing shoulder strength. J Shoulder Elb Surg 11(1):33–39
Kirk RE (1995) Experimental design: procedures for the behavioral sciences. Brooks/Cole, Pacific Grove
McGraw KO, Wong SP (1996) Forming inferences about some intraclass correlation coefficients. Psychol Methods 1(1):30–46
Müller R, Büttner P (1994) A critical discussion of intraclass correlation coefficients. Stat Med 13(23–24):2465–2476
Perisic I, Rosner B (1999) Comparisons of measures of interclass correlations: the general case of unequal group size. Stat Med 18(12):1451–1466
Rousson V, Gasser T, Seifert B (2003) Confidence intervals for intraclass correlation in inter-rater reliability. Scand J Stat 30(3):617–624
SAS Institute Inc. (2009) SAS/STAT® 9.2 user’s guide, 2nd ed. SAS Institute, Cary
SAS Institute Inc. (2011) SAS® 9.2 language reference: dictionary, 4th ed. SAS Institute, Cary
Satterthwaite PE (1946) An approximate distribution of estimates of variance components. Biometrics 2(6):110–114
Shavelson RJ, Webb NM (1991) Generalizability theory. SAGE, Newbury Park
Shrout PE, Fleiss J (1979) Intraclass correlation: uses in assessing rater reliability. Psychol Bull 86(2):420–428
St. Laurent RT (1998) Evaluating agreement with a gold standard in method comparison studies. Biometrics 54(2):537–545
Tian L, Cappelleri JC (2004) A new approach for interval estimation and hypothesis testing of a certain intraclass correlation coefficient: the generalized variable method. Stat Med 23(13):2125–2135
Wong SP, McGraw KO (1999) Confidence intervals and F tests for intraclass correlations based on three-way random effects models. Educ Psychol Meas 59(2):270–288
Zhou H, Muellerleile P, Ingram D, Wong SP (2011) Confidence intervals and F tests for intraclass correlation coefficients based on three-way mixed effects models. J Educ Behav Stat 36(5):638–671
Zou KH, McDermott MP (1999) Higher-moment approaches to approximate interval estimation for a certain intraclass correlation coefficient. Stat Med 18(15):2051–2061
Acknowledgments
This chapter was presented as an invited oral presentation on innovative methods in multivariate analysis at the International Chinese Statistical Association and the International Society for Biopharmaceutical Statistics Joint Statistical Conference, Bethesda, MD, USA June 9–12, 2013. The authors thank Kelly Zou (Pfizer Inc) for organizing the invited session. We also thank Tom Abernathy (Pfizer Inc) for help with execution of the simulation program.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Cappelleri, J., Ting, N. (2015). Evaluation of a Confidence Interval Approach for Relative Agreement in a Crossed Three-Way Random Effects Model. In: Chen, Z., Liu, A., Qu, Y., Tang, L., Ting, N., Tsong, Y. (eds) Applied Statistics in Biomedicine and Clinical Trials Design. ICSA Book Series in Statistics. Springer, Cham. https://doi.org/10.1007/978-3-319-12694-4_23
Download citation
DOI: https://doi.org/10.1007/978-3-319-12694-4_23
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-12693-7
Online ISBN: 978-3-319-12694-4
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)