Abstract
Recent literature places more emphasis on assessment comments rather than relying solely on scores. Both are variable, however, emanating from assessment judgements. One established source of variability is “contrast effects”: scores are shifted away from the depicted level of competence in a preceding encounter. The shift could arise from an effect on the range-frequency of assessors’ internal scales or the salience of performance aspects within assessment judgments. As these suggest different potential interventions, we investigated assessors’ cognition by using the insight provided by “clusters of consensus” to determine whether any change in the salience of performance aspects was induced by contrast effects. A dataset from a previous experiment contained scores and comments for 3 encounters: 2 with significant contrast effects and 1 without. Clusters of consensus were identified using F-sort and latent partition analysis both when contrast effects were significant and non-significant. The proportion of assessors making similar comments only significantly differed when contrast effects were significant with assessors more frequently commenting on aspects that were dissimilar with the standard of competence demonstrated in the preceding performance. Rather than simply influencing range-frequency of assessors’ scales, preceding performances may affect salience of performance aspects through comparative distinctiveness: when juxtaposed with the context some aspects are more distinct and selectively draw attention. Research is needed to determine whether changes in salience indicate biased or improved assessment information. The potential should be explored to augment existing benchmarking procedures in assessor training by cueing assessors’ attention through observation of reference performances immediately prior to assessment.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Alves De Lima, A., Barrero, C., Baratta, S., Castillo Costa, Y., Bortman, G., Carabajales, J., et al. (2007). Validity, reliability, feasibility and satisfaction of the mini-clinical evaluation exercise (mini-CEX) for cardiology residency training. Medical Teacher, 29(8), 785–790.
Cantillon, P., & Sargeant, J. (2008). Giving feedback in clinical settings. BMJ, 337(nov10_2), a1961.
Carifio, J., & Perla, R. (2008). Resolving the 50-year debate around using and misusing Likert scales. Medical Education, 42(12), 1150–1152.
Cook, D. A., Beckman, T. J., Mandrekar, J. N., & Pankratz, V. S. (2010). Internal structure of mini-CEX scores for internal medicine residents: Factor analysis and generalizability. Advances in Health Sciences Education, 15(5), 633–645.
Downing, S. M. (2004). Reliability: On the reproducibility of assessment data. Medical Education, 38(9), 1006–1012.
Downing, S. M. (2005). Threats to the validity of clinical teaching assessments: What about rater error? Medical Education, 39(4), 353–355.
Ende, J. (1983). Feedback in clinical medical education. JAMA: The Journal of the American Medical Association, 250(6), 777–781.
Fiske, S. T., & Cox, M. G. (1979). Person concepts: The effect of target familiarity and descriptive purpose on the process of describing others. Journal of Personality, 47(1), 136–161.
Gauthier, G., St-Onge, C., & Tavares, W. (2016). Rater cognition: Review and integration of research findings. Medical Education, 50(5), 511–522.
Gingerich, A., Kogan, J., Yeates, P., Govaerts, M., & Holmboe, E. (2014a). Seeing the ‘black box’ differently: Assessor cognition from three research perspectives. Medical Education, 48(11), 1055–1068.
Gingerich, A., Ramlo, S. E., van der Vleuten, C. P. M., Eva, K. W., & Regehr, G. (2017). Inter-rater variability as mutual disagreement: Identifying raters’ divergent points of view. Advances in Health Sciences Education, 22(4), 819–838.
Gingerich, A., van der Vleuten, C. P., Eva, K. W., & Regehr, G. (2014b). More consensus than idiosyncrasy: Categorizing social judgments to examine variability in mini-CEX ratings. Academic Medicine, 89(11), 1510–1519.
Ginsburg, S., van der Vleuten, C., Eva, K., & Lingard, L. (2016). Hedging to save face: A linguistic analysis of written comments on in-training evaluation reports. Advances in Health Sciences Education, 21(1), 175–188.
Higgins, E. T. (1996). Activation: Accessibility, and salience. In E. T. Higgins & A. W. Kruglanski (Eds.), Social psychology: Handbook of basic principles (pp. 133–168). New York: The Guilford Press.
Higgins, E. T. (2000). Social cognition: Learning about what matters in the social world. European Journal of Social Psychology, 30(1), 3–39.
Hill, F., Kendall, K., Galbraith, K., & Crossley, J. (2009). Implementing the undergraduate mini-CEX: A tailored approach at southampton university. Medical Education, 43(4), 326–334.
Kogan, J. R., Conforti, L., Bernabeo, E., Iobst, W., & Holmboe, E. (2011). Opening the black box of clinical skills assessment via observation: A conceptual model. Medical Education, 45(10), 1048–1060.
Lee, V., Brain, K., & Martin, J. (2017). Factors influencing mini-CEX rater judgments and their practical implications: A systematic literature review. Academic Medicine, 92(6), 880–887.
Margolis, M. J., Clauser, B. E., Cuddy, M. M., Ciccone, A., Mee, J., Harik, P., et al. (2006). Use of the mini-clinical evaluation exercise to rate examinee performance on a multiple-station clinical skills examination: A validity study. Academic Medicine: Journal of the Association of American Medical Colleges, 81(10 Suppl), S56–S60.
Miller, D. M., Wiley, D. E., & Wolfe, R. G. (1986). Categorization methodology: An approach to the collection and analysis of certain classes of qualitative information. Multivariate Behavioral Research, 21(2), 135.
Mohr, C. D., & Kenny, D. A. (2006). The how and why of disagreement among perceivers: An exploration of person models. Journal of Experimental Social Psychology, 42(3), 337–349.
Parducci, A., & Perrett, L. F. (1971). Category rating scales: Effects of relative spacing and frequency of stimulus values. Journal of Experimental Psychology, 89(2), 427.
Park, B., Dekay, M. L., & Kraus, S. (1994). Aggregating social behavior into person models: Perceiver-induced consistency. Journal of Personality and Social Psychology, 66(3), 437–459.
Ramani, S., & Krackov, S. K. (2012). Twelve tips for giving feedback effectively in the clinical environment. Medical Teacher, 34(10), 787–791.
Stevens, S. S. (1946). On the theory of scales of measurement. Science, 103(2684), 677–680.
Stewart, N., Brown, G. D. A., & Chater, N. (2005). Absolute identification by relative judgment. Psychological Review, 112(4), 881–911.
St-Onge, C., Chamberland, M., Lévesque, A., & Varpio, L. (2016). Expectations, observations, and the cognitive processes that bind them: Expert assessment of examinee performance. Advances in Health Sciences Education, 21(3), 627–642.
Tavares, W., Ginsburg, S., & Eva, K. W. (2016). Selecting and simplifying: Rater performance and behavior when considering multiple competencies. Teaching and Learning in Medicine, 28(1), 41–51.
van der Vleuten, C., Schuwirth, L., Driessen, E., Dijkstra, J., Tigelaar, D., Baartman, L., et al. (2012). A model for programmatic assessment fit for purpose. Medical Teacher, 34(3), 205–214.
Weller, J., Jones, A., Merry, A., Jolly, B., & Saunders, D. (2009). Investigation of trainee and specialist reactions to the mini-clinical evaluation exercise in anaesthesia: Implications for implementation. British Journal of Anaesthesia, 103(4), 524–530.
Wiley, D. E. (1967). Latent partition analysis. Psychometrika, 32(2), 183–193.
Yeates, P., Cardell, J., Byrne, G., & Eva, K. W. (2015a). Relatively speaking: Contrast effects influence assessors’ scores and narrative feedback. Medical Education, 49(9), 909–919.
Yeates, P., Moreau, M., & Eva, K. (2015b). Are examiners’ judgments in osce-style assessments influenced by contrast effects? Academic Medicine, 90(7), 975–980.
Yeates, P., O’Neill, P., Mann, K., & Eva, K. W. (2012). Effect of exposure to good vs poor medical trainee performance on attending physician ratings of subsequent performances. JAMA: Journal of the American Medical Association, 308(21), 2226–2232.
Yeates, P., O’Neill, P., Mann, K., & Eva, K. W. (2013a). Seeing the same thing differently: Mechanisms that contribute to assessor differences in directly-observed performance assessments. Advances in Health Sciences Education, 18(3), 325–341.
Yeates, P., O’Neill, P., Mann, K., & Eva, W. K. (2013b). ‘You’re certainly relatively competent’: Assessor bias due to recent experiences. Medical Education, 47(9), 910–922.
Acknowledgements
We are grateful for partial funding provided by a Northern Medical Program seed grant.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Ethical approval
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional research ethics boards.
Rights and permissions
About this article
Cite this article
Gingerich, A., Schokking, E. & Yeates, P. Comparatively salient: examining the influence of preceding performances on assessors’ focus and interpretations in written assessment comments. Adv in Health Sci Educ 23, 937–959 (2018). https://doi.org/10.1007/s10459-018-9841-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10459-018-9841-2