Abstract
Background
Attentional bias to threat is a fundamental transdiagnostic component and potential vulnerability factor for internalizing psychopathologies. However, the measurement of attentional bias, such as traditional scores from the dot-probe paradigm, evidence poor reliability and do not measure intra-individual variation in attentional bias.
Methods
The present study examined, in three independent samples, the psychometric properties of a novel attentional bias (AB) scoring method of the dot-probe task based on responses to individual trials. For six AB scores derived using the response-based approach, we assessed the internal consistency, test–retest reliability, familial associations, and external validity (using Social Anxiety Disorder, a disorder strongly associated with attentional bias to threatening faces).
Results
Compared to traditional AB scores, response-based scores had generally better internal consistency (range of Cronbach’s alphas: 0.68–0.92 vs. 0.41–0.71), higher test–retest reliabilities (range of Pearson’s correlations: 0.26–0.77 vs. − 0.05 to 0.35), and were more strongly related in family members (range of ICCs: 0.11–0.27 vs. 0–0.05). Furthermore, three response-based scores added incremental validity beyond traditional scores and gender in the external validators of current and lifetime Social Anxiety Disorder.
Conclusions
Findings indicate that response-based AB scores from the dot-probe task have better psychometric properties than traditional scores.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
We explored whether AB would vary under ‘threat’ and ‘safe’ contexts (order of safe and threat were counterbalanced). During the threatening context, participants heard random presentations of a woman screaming or nails scra** on slate (see Neumann et al. 2008; who used these sounds as unconditioned stimuli in Pavlovian conditioning). The traditional and response-based measures of AB were comparable under threat vs. safe contexts (all p’s > 0.16) as were the reliabilities. Thus, for ease of presentation of results and to increase power, all analyses combined trials from threat and safe contexts.
References
Amir, N., Elias, J., Klumpp, H., & Przeworski, A. (2003). Attentional bias to threat in social phobia: Facilitated processing of threat or difficulty disengaging attention from threat? Behaviour Research and Therapy, 41, 1325–1335. https://doi.org/10.1016/S0005-7967(03)00039-1.
Amir, N., Prouvost, C., & Kuckertz, J. M. (2012). Lack of a benign interpretation bias in social anxiety disorder. Cognitive Behaviour Therapy, 41(2), 119–129. https://doi.org/10.1080/16506073.2012.662655.
Bantin, T., Stevens, S., Gerlach, A. L., & Hermann, C. (2016). What does the facial dot-probe task tell us about attentional processes in social anxiety? A systematic review. Journal of Behavior Therapy and Experimental Psychiatry, 50, 40–51. https://doi.org/10.1016/j.jbtep.2015.04.009.
Bar-Haim, Y., Lamy, D., Pergamin, L., Bakermans-Kranenburg, M. J., & van Ijzendoorn, M. H. (2007). Threat-related attentional bias in anxious and nonanxious individuals: a meta-analytic study. Psychological Bulletin, 133(1), 1–24. https://doi.org/10.1037/0033-2909.133.1.1.
Cisler, J. M., & Koster, E. H. W. (2010). Mechanisms of attentional biases towards threat in anxiety disorder: An integrative review. Clinical Psychology Review, 30(2), 1–29. https://doi.org/10.1016/j.cpr.2009.11.003.Mechanisms.
Clark, D., & Wells, A. (1995). A cognitive model of social phonia. In Social phobia (p. 69). New York.
Clarke, P. J. F., MacLeod, C., & Guastella, A. J. (2013). Assessing the role of spatial engagement and disengagement of attention in anxiety-linked attentional bias: A critique of current paradigms and suggestions for future research directions. Anxiety Stress and Co**, 26(1), 1–19.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences, 2nd ed. Hillsdale, NJ: Erlbaum.
Crawford, J., Cayley, C., Lovibond, P. F., Wilson, P. H., & Hartley, C. (2011). Percentile norms and accompanying interval estimates from an australian general adult population sample for self-report mood scales (BAI, BDI, CRSD, CES-D, DASS, DASS-21, STAI-X, STAI-Y, SRDS, and SRAS). Australian Psychologist, 46(1), 3–14. https://doi.org/10.1111/j.1742-9544.2010.00003.x.
Egloff, B., & Hock, M. (2003). Assessing attention allocation toward threat-related stimuli: A comparison of the emotional Stroop task and the attentional probe task. Personality and Individual Differences, 35, 475–483.
Evans, T. C., Bar-Haim, Y., Fox, N. A., Pine, D. S., & Britton, J. C. (2020). Neural mechanisms underlying heterogeneous expression of threat-related attention in social anxiety. Behaviour Research and Therapy, 132, 103657. https://doi.org/10.1016/j.brat.2020.103657.
Evans, T. C., & Britton, J. C. (2018). Improving the psychometric properties of dot-probe attention measures using response-based computation. Journal of Behavior Therapy and Experimental Psychiatry. https://doi.org/10.1016/j.jbtep.2018.01.009.
Evans, T. C., Walukevich, K. A., & Britton, J. C. (2016). Vigilance-avoidance and disengagement are differentially associated with fear and avoidant behaviors in social anxiety. Journal of Affective Disorders, 199, 124–131. https://doi.org/10.1016/j.jad.2016.04.003.
Evans, T. C., Walukevich, K. A., Seager, I., & Britton, J. C. (2018). A psychometric comparison of anxiety-relevant attention measures. Anxiety, Stress, & Co**, 31(5), 539–554. https://doi.org/10.1080/10615806.2018.1489536.
Eysenck, M. W., Santos, R., & Calvo, M. G. (2007). Anxiety and cognitive performance: Attentional control theory. Emotion, 7(2), 336–353. https://doi.org/10.1037/1528-3542.7.2.336.
First, M. B., Williams, J. B. W., Karg, R. S., & Spitzer, R. L. (2015). Structured clinical interview for DSM-5: Research version (SCID-5 for DSM-5, research version; SCID-5-RV). Arlington: American Psychiatric Association.
Fox, E. (2002). Processing emotional facial expressions: The role of anxiety and awareness. Cognitive, Affective & Behavioral Neuroscience, 2(1), 52–63. https://doi.org/10.1080/02699930143000527.
Gorka, S. M., Hee, D., Lieberman, L., Mittal, V. A., Phan, K. L., & Shankman, S. A. (2016). Reactivity to uncertain threat as a familial vulnerability factor for alcohol use disorder. Psychological Medicine, 46(16), 3349–3358. https://doi.org/10.1017/S0033291716002415.
Hakamata, Y., Lissek, S., Bar-Haim, Y., Britton, J. C., Fox, N., Leibenluft, E., et al. (2010). Attention bias modification treatment: A meta-analysis towards the establishment of novel treatment for anxiety. Biological Psychiatry, 76(October 2009), 211–220. https://doi.org/10.1007/s11103-011-9767-z.Plastid.
Hettema, J. M., Neale, M. C., & Kendler, K. S. (2001). A review and meta-analysis of the genetic epidemiology of anxiety disorders. American Journal of Psychiatry, 158(10), 1568–1578. https://doi.org/10.1176/appi.ajp.158.10.1568.
Iacoviello, B. M., Wu, G., Abend, R., Murrough, J. W., Feder, A., Fruchter, E., et al. (2014). Attention bias variability and symptoms of posttraumatic stress disorder. Journal of Traumatic Stress, 27, 232–239. https://doi.org/10.1002/jts.
Kappenman, E. S., Farrens, J. L., Luck, S. J., & Proudfit, G. H. (2014). Behavioral and ERP measures of attentional bias to threat in the dot-probe task: Poor reliability and lack of correlation with anxiety. Frontiers in Psychology, 5(December), 1–9. https://doi.org/10.3389/fpsyg.2014.01368.
Kappenman, E. S., Macnamara, A., & Proudfit, G. H. (2015). Electrocortical evidence for rapid allocation of attention to threat in the dot-probe task. Social Cognitive and Affective Neuroscience, 10, 577–583. https://doi.org/10.1093/scan/nsu098.
Kendler, K. S. (2006). Reflections on the relationship between psychiatric genetics and psychiatric nosology. American Journal of Psychiatry, 163(July), 1138–1146.
Klein, D. N., Shankman, S. A., Lewinsohn, P. M., Rohde, P., & Seeley, J. R. (2004). Family study of chronic depression in a community sample of young adults. American Journal of Psychiatry, 161(4), 646–653. https://doi.org/10.1176/appi.ajp.161.4.646.
Kruijt, A. W., Field, A. P., & Fox, E. (2016). Capturing dynamics of biased attention: Are new attention variability measures the way forward? PLoS ONE, 11(11), e0166600.
Kruijt, A. W., Parsons, S., & Fox, E. (2019). A meta-analysis of bias at baseline in RCTs of attention bias modification: No evidence for dot-probe bias towards threat in clinical anxiety and PTSD. Journal of Abnormal Psychology, 128(6), 563–573. https://doi.org/10.1037/abn0000406.
Lavie, N. (1995). Perceptual load as a necessary condition for selective attention. Journal of Experimental Psychology: Human Perception and Performance, 21(3), 451–468. https://doi.org/10.1037//0096-1523.21.3.451.
Lovibond, S., & Lovibond, P. (1995). Manual for the depression anxiety stress scales. Sydney: The Psychology Foundation of Australia Inc.
MacLeod, C., & Hagan, R. (1992). Individual differences in the selective processing of threatening information, and emotional responses to a stressful life event. Behaviour Research and Therapy, 30(2), 151–161. https://doi.org/10.1016/0005-7967(92)90138-7.
MacLeod, C., Mathews, A., & Tata, P. (1986). Attentional bias in emotional disorders. Journal of Abnormal Psychology, 95(1), 15–20. https://doi.org/10.1037/0021-843X.95.1.15.
MacLeod, C., Rutherford, E., Campbell, L., Ebsworthy, G., & Holker, L. (2002). Selective attention and emotional vulnerability: Assessing the causal basis of their association through the experimental manipulation of attentional bias. Journal of Abnormal Psychology, 111(1), 107–123. https://doi.org/10.1037/0021-843X.111.1.107.
Mansell, W., Clark, D. M., Ehlers, A., Chen, Y., Mansell, W., Clark, D. M., et al. (1999). Social anxiety and attention away from emotional faces. Cognition and Emotion, 13(6), 673–690. https://doi.org/10.1080/026999399379032.
Mathews, A., Ridgeway, V., & Williamson, D. A. (1996). Evidence for attention to threatening stimuli in depression. Behaviour Research and Therapy, 34(9), 695–705. https://doi.org/10.1016/0005-7967(96)00046-0.
McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30–46.
Mittlböck, M., & Schemper, M. (1996). Explained variation for logistic regression. Statistics in Medicine, 15(19), 1987–1997.
Mogg, K., & Bradley, B. P. (1999). Orienting of attention to threatening facial expressions presented under conditions of restricted awareness. Cognition and Emotion, 13(6), 713–740. https://doi.org/10.1080/026999399379050.
Mogg, K., Philippot, P., & Bradley, B. P. (2004). Selective attention to angry faces in clinical social phobia. Journal of Abnormal Psychology, 113(1), 160.
National Advisory Mental Health Council Workgroup on Tasks and Measures for Research Domain Criteria. (2016). Behavioral assessment methods for RDoC constructs. Retrieved from https://www.nimh.nih.gov/about/advisory-boards-and-groups/namhc/reports/behavioral-assessment-methods-for-rdoc-constructs.shtml.
Neumann, D. L., & Waters, A. M. (2006). The use of an unpleasant sound as an unconditional stimulus in a human aversive Pavlovian conditioning procedure. Biological Psychology, 73, 175–185. https://doi.org/10.1016/j.biopsycho.2006.03.004.
Neumann, D. L., Waters, A. M., & Westbury, H. R. (2008). The use of an unpleasant sound as the unconditional stimulus in aversive Pavlovian conditioning experiments that involve children and adolescent participants. Behavior Research Methods, 40(2), 622–625. https://doi.org/10.3758/BRM.40.2.622.
Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York: McGraw-Hill.
Price, R. B., Kuckertz, J. M., Siegle, G. J., Ladouceur, C. D., Silk, J. S., Ryan, N. D., et al. (2015). Empirical recommendations for improving the stability of the Dot-Probe Task in clinical research. Psychological Assessment, 27(2), 365–376. https://doi.org/10.1037/pas0000036.
Robins, E., & Guze, S. B. (1970). Establishment of diagnostic validity in psychiatric illness: Its application to schizophrenia. American Journal of Psychiatry, 126(7), 983–987.
Rodebaugh, T. L., Scullin, R. B., Langer, J. K., Dixon, D. J., Huppert, J. D., Bernstein, A., et al. (2016). Unreliability as a threat to understanding psychopathology: The cautionary tale of attentional bias. Journal of Abnormal Psychology, 125(6), 840–851. https://doi.org/10.1037/abn0000184.
Schmukle, S. C. (2005). Unreliability of the dot probe task. European Journal of Personality, 19(7), 595–605. https://doi.org/10.1002/per.554.
Shankman, S. A., & Klein, D. N. (2003). The relation between depression and anxiety: an evaluation of the tripartite, approach-withdrawal and valence-arousal models. Clinical Psychology Review, 23, 605–637. https://doi.org/10.1016/S0272-7358(03)00038-2.
Shankman, S. A., Lerner, D., Funkhouser, C. J., Klein, D. N., & Davila, J. (2018). Reliability and validity of severity dimensions of psychopathology assessed using the Structured Clinical Interview for DSM - 5 (SCID). International Journal of Methods in Psychiatry Research. https://doi.org/10.1002/mpr.1590.
Shechner, T., Britton, J. C., Pérez-edgar, K., & Bar-haim, Y. (2012). Attention biases, anxiety, and development: Toward or away from threats or rewards? Depress Anxiety, 29(4), 282–294. https://doi.org/10.1002/da.20914.Attention.
Shrout, P. E. (1998). Measurement reliability and agreement in psychiatry. Statistical Methods in Medical Research, 7(3), 301–317. https://doi.org/10.1177/096228029800700306.
Spielberg, J. M., De Leon, A. A., Bredemeier, K., Heller, W., Engels, A. S., Warren, S. L., et al. (2013). Anxiety type modulates immediate versus delayed engagement of attention-related brain regions. Brain and Behavior, 3(5), 532–551. https://doi.org/10.1002/brb3.157.
Staugaard, S. R. (2009). Reliability of two versions of the dot-probe task using photographic faces. Psychology Science Quarterly, 51(3), 339–350.
Tottenham, N., Tanaka, J., Leon, A., McCarry, T., Nurse, M., Hare, T., et al. (2009). The NimStim set of facial expressions: Judgments from untrained research participants. Psychiatry Research, 168(3), 242–249. https://doi.org/10.1016/j.psychres.2008.05.006.The.
Waechter, S., Nelson, A. L., Wright, C., Hyatt, A., & Oakman, J. (2014). Measuring attentional bias to threat: Reliability of dot probe and eye movement indices. Cognitive Therapy and Research, 38(3), 313–333. https://doi.org/10.1007/s10608-013-9588-2.
Weinberg, A., Liu, H., Hajcak, G., & Shankman, S. A. (2016). Blunted neural response to rewards as a vulnerability factor for depression: Results from a family study. Journal of Abnormal Psychology, 124(4), 878–889. https://doi.org/10.1037/abn0000081.Blunted.
Zubin, J., & Spring, B. (1977). Vulnerability: A new view of schizophrenia. Journal of Abnormal Psychology, 86(2), 103–126. https://doi.org/10.1037//0021-843X.86.2.103.
Zvielli, A., Bernstein, A., & Koster, E. H. W. (2015). Temporal dynamics of attentional bias. Clinical Psychological Science, 3(5), 772–788. https://doi.org/10.1177/2167702614551572.
Acknowledgements
This work was supported by the National Institute of Mental Health Grant R01 MH098093 and National Institute on Drug Abuse Grant R21 DA045285 awarded to Dr. Shankman.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interest
Emily E. E. Meissel, Huiting Liu, Elizabeth S. Stevens, Travis C. Evans, Jennifer C. Britton, Allison M. Letkiewicz and Stewart A. Shankman declare that they have no conflict of interest.
Informed Consent
All three studies were approved by the University of Illinois at Chicago International Review Board and were performed in accordance with the ethical standards as laid down in the latest version of the Declaration of Helsinki.
Animal Rights Statements
There were no non-human animals involved in this research.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Meissel, E.E.E., Liu, H., Stevens, E.S. et al. The Reliability and Validity of Response-Based Measures of Attention Bias. Cogn Ther Res 46, 146–160 (2022). https://doi.org/10.1007/s10608-021-10212-w
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10608-021-10212-w