Abstract
As assessments are used in an increasingly multicultural and connected world, there is a growing need to verify that they are equally valid across different populations. More specifically, when using hiring assessments to select people for jobs, it is important to corroborate that direct comparisons of individuals from different populations are valid, leading to fair and accurate hires. Populations differ in many interesting ways, but in this chapter, we examined how cultural group differences affect assessment behavior. Thus, we set out to disentangle the effects of location and currency, as elements of cultural behavior, on constructs used in hiring assessments: fairness, altruism, and decision-making speed. These constructs are measured in our gamified implementation of the Trust Game (Berg et al. (Games Econ Behav 10:122–142, 1995)) and Dictator Game (Savin and Sefton (Games Econ Behav 6:347–369, 1994)). We had data from job candidates in many world regions, who responded in various languages and game money currencies. Using this, we tested the factorial invariance of the measures from the two games. We compared large groups across different regions (controlling for language and currency), namely the United States and China, and across different currencies (controlling for language and region), specifically euros and reales. While the general factor structure held across all groups, we found differences in the observed variables, which varied by group. The findings highlight the importance of considering cultural influences when interpreting assessment results and underscore the significance of measurement invariance in promoting fairness and accuracy in hiring processes.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
To avoid capitalization on chance in model selection, apart from their application to the measurement invariance test samples, all of the several models under consideration were compared for fit on three non-overlap** samples, each comprising \(N=34{,}997\) or \(N=34{,}998\) English language response patterns from the United States. Based primarily on RMSEA as a selection criterion, the same best model emerged each time. RMSEA was chosen as the focus because, among the fit indices considered, its value was least satisfactory. We acknowledge the possibility that other, untested models may fit our data.
- 2.
- 3.
In our studies the choice of reference and focal groups is completely arbitrary.
- 4.
Testing the correlation constraint, as opposed to the covariance constraint, in a model with non-standardized observations was accomplished by simultaneously fitting both the factor model as described and a linear transformation of that model into standardized form; this can be accomplished using lavaan’s regression syntax with many parameters constrained.
- 5.
We did not attempt to evaluate strict invariance, wherein unique variances are constrained.
- 6.
In data for a similar study (Li et al., this volume), we also found that around 10% of Spanish speakers in six different countries take this money back.
References
Berg, J., Dickhaut, J., & McCabe, K. (1995). Trust, reciprocity, and social history. Games and Economic Behavior, 10, 122–142.
Brülhart, M. & Usunier, J. C. (2012). Does the trust game measure trust? Economics Letters, 115, 20–23.
Carpenter, J., Connolly, C., & Myers, C. K. (2008). Altruistic behavior in a representative dictator experiment. Experimental Economics, 11, 282–298.
Embretson, S. E. & Wetzel, C. D. (1987). Component latent trait models for paragraph comprehension tests. Applied Psychological Measurement, 11(2), 175–193.
Forsythe, R., Horowitz, J. L., Savin, N. E., & Sefton, M. (1994). Fairness in simple bargaining experiments. Games and Economic Behavior, 6, 347–369.
Meredith, W. (1993). Measurement invariance, factor analysis, and factorial invariance. Psychometrika, 58, 525–543.
Neyman, J. & Pearson, E. S. (1928). On the use and interpretation of certain test criteria for purposes of statistical inference: Part I. Biometrika, 20A, 175–240.
Ployhart, R. E. (2006). The predictor response process model. In J. W. R. Ployhart (Ed.), Situational judgment tests: Theory, measurement, and application (pp. 83–106). Erlbaum.
Schwarz, G. E. (1978). Estimating the dimension of a model. Annals of Mathematical Statistics, 6, 461–464.
Thissen, D. & Thissen-Roe, A. (2020). Factor score estimation from the perspective of item response theory. In M. Wiberg, D. Molenaar, J. González, U. Bockenhold, & J.-S. Kim (Eds.), Quantitative Psychology: 84th Annual Meeting of the Psychometric Society, Santiago, Chile, 2019 (pp. 171–184). Springer.
Thurstone, L. L. (1935). The vectors of mind. University of Chicago Press.
Vandenberg, R. J. & Lance, C. E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3, 4–70.
Wilks, S. S. (1938). The large-sample distribution of the likelihood ratio for testing composite hypotheses. Annals of Mathematical Statistics, 9, 60–62.
Acknowledgements
We would like to extend a big thank you to Dave Thissen. Furthermore, we would like to thank Simon Moon and Hao Wu, as well as everyone at IMPS 2023. Finally, we want to thank the leadership team at Harver, especially Ben Porr, for supporting our research.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Kleinbort, A., Li, A., Szary, J., Thissen-Roe, A. (2024). Global Validity of Assessments: Location and Currency Effects. In: Hwang, H., Wu, H., Sweet, T. (eds) Quantitative Psychology. IMPS 2023. Springer Proceedings in Mathematics & Statistics, vol 452. Springer, Cham. https://doi.org/10.1007/978-3-031-55548-0_31
Download citation
DOI: https://doi.org/10.1007/978-3-031-55548-0_31
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-55547-3
Online ISBN: 978-3-031-55548-0
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)