SpringerLink
Log in

The role of holistic review in diversifying graduate medical education: a systematic review and meta-analysis

  • Review
  • Published:
Global Surgical Education - Journal of the Association for Surgical Education Aims and scope Submit manuscript

Abstract

Purpose

Diversity is crucial to the success of healthcare teams and inclusive patient care. The emphasis on traditional academic metrics has served as a barrier to the diversification of graduate medical education (GME) programs. Holistic review, defined by the Association of American Medical Colleges as an emphasis on experiences rather than academic metrics, has been proposed as a solution to improve diversity in medical education. However, skepticism about the implementation of holistic review and a lack of data to support that it improves diversity in GME exist. We, therefore, performed a systematic review and meta-analysis aiming to identify the components of holistic review described by programs employing it and quantify its impact on diversity. We hypothesized that programs implementing holistic review would have increased odds of interviewing and matriculating those that are underrepresented in medicine (URiM) and women when compared to those utilizing traditional applicant review.

Methods

PubMed and Embase were searched from inception to February 2023 for articles studying holistic review in United States GME programs. Articles were included if they described components of holistic review and/or compared the diversity of traditionally reviewed cohorts to those holistically reviewed. Two authors performed title/abstract screening; any disagreements were adjudicated by a third reviewer. Eligible studies were submitted to full-text screening. The association between holistic review and the proportion of URiM students interviewed/matriculated was assessed using random-effects meta-analysis. In addition, components of holistic review were categorized and described.

Results

Of 201 screened abstracts, 21 articles were included. Four features of holistic review were consistently described by the included studies: (1) incorporating a mission-guided approach to applicant selection, (2) mitigating bias, (3) de-emphasizing academic metrics, (4) commitment to diversity, equity, and inclusion. Six studies compared proportions of URiM students interviewed using holistic versus traditional applicant review; holistic review was associated with increased odds of interviewing URiM students at these programs (pooled OR 2.30, 95% CI 1.19, 4.44; I2 = 85.66%). Five studies reported data on URiMs matriculated into their programs using holistic versus traditional applicant review; holistic review was also associated with increased pooled odds of URiM students matriculating into these programs (pooled OR 2.73, 95% CI 1.25, 5.95; I2 = 22.10%). There was insufficient data to complete meta-analysis on the effect of holistic review on proportions of women interviewed or matriculated into residency programs. The GRADE quality of evidence to support the implementation of holistic review to improve diversity in GME programs based upon the studies included in this review is moderate.

Conclusions

Holistic review is a feasible and effective way to increase diversity in GME programs. This systematic review outlines the different methods by which programs can screen, interview, and rank applicants with a mission-driven lens and increased emphasis on experiences and attributes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Gomez LE, Bernet P. Diversity improves performance and outcomes. J Natl Med Assoc. 2019;111(4):383–92.

    CAS  PubMed  Google Scholar 

  2. Saha S, Taggart SH, Komaromy M, Bindman AB. Do patients choose physicians of their own race? Health Aff. 2000;19:76–83.

    Article  CAS  Google Scholar 

  3. Jetty A, Jabbarpour Y, Pollack J, Huerto R, Woo S, Petterson S. Patient-physician racial concordance associated with improved healthcare use and lower healthcare expenditures in minority populations. J Racial Ethn Health Disparities. 2022;9(1):68–81.

    Article  PubMed  Google Scholar 

  4. Malhotra J, Rotter D, Tsui J, Llanos AAM, Balasubramanian BA, Demissie K. Impact of patient-provider race, ethnicity, and gender concordance on cancer screening: findings from medical expenditure panel survey. Cancer Epidemiol Biomarkers Prev. 2017;26(12):1804–11.

    Article  PubMed  Google Scholar 

  5. Underrepresented in Medicine Definition. AAMC. https://www.aamc.org/what-we-do/equity-diversity-inclusion/underrepresented-in-medicine. Accessed April 24, 2024.

  6. Kim Y, Kassam AF, McElroy IE, et al. The current status of the diversity pipeline in surgical training. Am J Surg. 2022;224(1 Pt B):250–6.

    Article  PubMed  Google Scholar 

  7. Abelson JS, Symer MM, Yeo HL, et al. Surgical time out: Our counts are still short on racial diversity in academic surgery. Am J Surg. 2018;215(4):542–8.

    Article  PubMed  Google Scholar 

  8. Columbus AB, Lu PW, Hill SS, Fields AC, Davids JS, Melnitchouk N. Factors associated with the professional success of female surgical department chairs: a qualitative study. JAMA Surg. 2020;155(11):1028–33.

    Article  PubMed  PubMed Central  Google Scholar 

  9. Roberts SE, Shea JA, Sellers M, Butler PD, Kelz RR. Pursing a career in academic surgery among African American medical students. Am J Surg. 2020;219(4):598–603.

    Article  CAS  PubMed  Google Scholar 

  10. Myers SP, Dasari M, Brown JB, et al. Effects of gender bias and stereotypes in surgical training: a randomized clinical trial. JAMA Surg. 2020;155(7):552–60.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Lucey CR, Saguil A. The consequences of structural racism on MCAT scores and medical school admissions: the past is prologue. Acad Med. 2020;95(3):351–6.

    Article  PubMed  Google Scholar 

  12. Conrad SS, Addams AN, Young GH. Holistic review in medical school admissions and selection: a strategic, mission-driven response to shifting societal needs. Acad Med. 2016;91(11):1472–4.

    Article  PubMed  Google Scholar 

  13. USMLE Step 1 Transition to Pass/Fail Only Score Reporting. https://www.usmle.org/usmle-step-1-transition-passfail-only-score-reporting. Accessed 26 Feb 2024.

  14. Chisholm LP, Drolet BC. USMLE step 1 scoring changes and the urology residency application process: program directors’ perspectives. Urology. 2020;145:79–82.

    Article  PubMed  Google Scholar 

  15. Artinian NT, Drees BM, Glazer G, et al. Holistic admission in the health professions: strategies for leaders. Coll Univ. 2017;92:65–86.

    PubMed  PubMed Central  Google Scholar 

  16. Aziz H, Khan S, Rocque B, Javed MU, Sullivan ME, Cooper JT. Selecting the next generation of surgeons: general surgery program directors and coordinators perspective on USMLE changes and holistic approach. World J Surg. 2021;45(11):3258–65.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Althans AR, Byrd T, Suppok R, Lee KK, Rosengart MR, Myers SP. Impact of holistic review on diversity of interviewed and matriculating residents in graduate medical education: a systematic review protocol. BMJ Open. 2023;13(7): e074118.

    Article  PubMed  PubMed Central  Google Scholar 

  18. Otten R, Vries S. Amsterdam efficient deduplication method. Zenodo. 2019.

  19. Bramer WM, Giustini D, de Jonge GB, Holland L, Bekhuis T. De-duplication of database search results for systematic reviews in EndNote. J Med Libr Assoc. 2016;104(3):240–3.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Higgins JP, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ. 2003;327:557–60.

    Article  PubMed  PubMed Central  Google Scholar 

  21. Sterne JA, Hernan MA, Reeves BC, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355: i4919.

    Article  PubMed  PubMed Central  Google Scholar 

  22. Siemieniuk R, Guyatt G. BMJ Best Practice: What is GRADE? https://bestpractice.bmj.com/info/us/toolkit/learn-ebm/what-is-grade/. Accessed 26 Feb 2024.

  23. Holistic Review in Medical School Admissions. https://students-residents.aamc.org/media/5256/download. Accessed 2 Jan 2024.

  24. McLemore L, Achrati, et al. Pathology-specific residency application scoring metric: An attempt to decrease bias in applicant selection. Lab Invest. 2021;101(1):349.

    Google Scholar 

  25. Bosslet GT, Carlos WG 3rd, Tybor DJ, et al. Multicenter validation of a customizable scoring tool for selection of trainees for a residency or fellowship program: the EAST-IST study. Ann Am Thorac Soc. 2017;14(4):517–23.

    Article  PubMed  Google Scholar 

  26. Nehemiah A, Roberts SE, Song Y, et al. Looking beyond the numbers: increasing diversity and inclusion through holistic review in general surgery recruitment. J Surg Educ. 2021;78(3):763–9.

    Article  PubMed  Google Scholar 

  27. Chi B, Krull, et al. Incorporating holistic review in recruitment in a physical medicine and rehabilitation residency program. Am J Phys Med Rehabil. 2022;101(9):859–63.

    Article  PubMed  Google Scholar 

  28. Jimenez RB, Pinnix CC, et al. Using holistic residency applicant review and selection in radiation oncology to enhance diversity and inclusion, an ASTRO SCAROP-ADROP-ARRO collaboration. Int J Radiat Oncol Biol Phys. 2023;116(2):334–47.

    Article  PubMed  Google Scholar 

  29. Oller D. Starting the conversation: analyzing responses to a single question engaging applicants in discussions of diversity on interview day. J Gen Intern Med. 2020;35(SUPPL 1):S768.

    Google Scholar 

  30. Aibana O, Swails JL, Flores RJ, Love L. Bridging the gap: holistic review to increase diversity in graduate medical education. Acad Med. 2019;94(8):1137–41.

    Article  PubMed  Google Scholar 

  31. DeRosa P, Takacs E, Wendt L, Tracy C. Effect of holistic review, interview blinding, and structured questions in resident selection: can we predict who will do well in a residency interview? Urology. 2023;173:41–7.

    Article  Google Scholar 

  32. Lee S, Geslani V, Koh S, Yu M, Loo L. Overcoming unconscious bias and rating errors in the residency selection process. J Gen Intern Med. 2018;33(2):279.

    Google Scholar 

  33. Sungar WG, Angerhofer C, et al. Implementation of holistic review into emergency medicine residency application screening to improve recruitment of underrepresented in medicine applicants. AEM Educ Train. 2021;5(Suppl 1):S10–8.

    PubMed  PubMed Central  Google Scholar 

  34. Barcelo NE, Shadravan S, Wells CR, et al. Reimagining merit and representation: promoting equity and reducing bias in GME through holistic review. Acad Psychiatry. 2021;45(1):34–42.

    Article  PubMed  Google Scholar 

  35. Williams C, Familusi O, et al. Representation matters: one urology residency program’s approach to increasing workforce diversity. Urology. 2022;174:28–34.

    Article  PubMed  Google Scholar 

  36. Thompson BN, Colbert K, et al. Practical strategies for underrepresented minority recruitment in general surgery residency. J Surg Educ. 2022;79(6):e130–6.

    Article  PubMed  Google Scholar 

  37. Alvarez A, Caretta W, et al. Holistic review and #match2021: Aligning screening with institutional mission, vision, and values. Western J Emerg Med. 2022;23(1.1):S56.

    Google Scholar 

  38. Iwai Y, Yu AYL, Daniels NC, et al. Racial, ethnic, and gender diversity among academic surgical leaders in the US. JAMA Surg. 2023;158(12):1328–34.

    Article  PubMed  Google Scholar 

  39. Lucey CR, Hauer KE, Boatright D, Fernandez A. Medical education’s wicked problem: achieving equity in assessment for medical learners. Acad Med. 2020;95(12S):S98–108.

    Article  PubMed  Google Scholar 

  40. Sottile E, Pasha AS, Ufomata E, et al. Affirmative action ends and the imperative to diversify medicine must intensify. J Gen Intern Med. 2023;39(5):863–6.

    Article  PubMed  Google Scholar 

  41. Henderson M, Green C, Chen C. What does it mean for medical school admissions to be socially accountable? AMA J Ethics. 2021;23:965–74.

    Article  Google Scholar 

  42. Gallegos M, Landry A, Alvarez A, et al. Holistic review, mitigating bias, and other strategies in residency recruitment for diversity, equity, and inclusion: an evidence-based guide to best practices from the council of residency directors in emergency medicine. West J Emerg Med. 2022;23(3):345–52.

    Article  PubMed  PubMed Central  Google Scholar 

  43. Schulz AE, Nussbaum JE, Loloi J, Sankin A, Abraham N. The impact of holistic review of urology residency applications on selection for interview during the COVID-19 pandemic. Urology. 2023;173:34–40.

    Article  PubMed  Google Scholar 

  44. Marbin J, Rosenbluth G, et al. Improving Diversity in Pediatric Residency Selection: Using an Equity Framework to Implement Holistic Review. J Grad Med Educ. 2021;13(2):195–200.

    Article  PubMed  PubMed Central  Google Scholar 

  45. Hemal K, Reghunathan M, et al. Diversity and inclusion: a review of effective initiatives in surgery. J Surg Educ. 2021;78(5):1500–15.

    Article  PubMed  Google Scholar 

  46. Caiado A, Lev R, et al. Improving Diversity Consciousness: Initiatives for Increasing Emergency Medicine Residency Diversity Recruitment. Western J Emerg Med. 2022;23(4.1):S17–8.

    Google Scholar 

  47. Nussbaum J, Loloi J, et al. Implementation and assessment of a holistic approach to urology residency applications. J Urol. 2021;206(SUPPL 3):e192.

    Google Scholar 

  48. Lupton K, Schaeffer, et al. Moving the needle: Implementing holistic reviewin internal medicine (IM) residency admissions. J General Intern Med. 2017;32(2):S693–4.

    Google Scholar 

  49. Swails JL, Adams S, Hormann M, Omoruyi E, Aibana O. Mission-based filters in the electronic residency application service: saving time and promoting diversity. J Grad Med Educ. 2021;13(6):785–94.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Funding

The authors did not receive support from any organization for the submitted work.

Author information

Authors and Affiliations

  1. Department of Surgery, University of Pittsburgh Medical Center, Presbyterian Hospital, 200 Lothrop Street, Pittsburgh, PA, 15213, USA

    Alison R. Althans, Tamara Byrd & Kenneth K. Lee

  2. Health Sciences Library System, University of Pittsburgh, M722 Alan Magee Scaife Hall, 3550 Terrace St, Pittsburgh, PA, 15261, USA

    Rachel Suppok

  3. Department of Surgery, Washington University in St. Louis, 4921 Parkview Pl, St. Louis, MO, 63110, USA

    Matthew R. Rosengart

  4. Division of Breast Surgery, Department of Surgery, Brigham and Women’s Hospital, 450 Brookline Avenue, Boston, MA, 02215, USA

    Sara P. Myers

Authors
  1. Alison R. Althans
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tamara Byrd
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rachel Suppok
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kenneth K. Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Matthew R. Rosengart
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Sara P. Myers
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alison R. Althans.

Ethics declarations

Conflict of interest

The authors have no relevant disclosures or conflicts of interest to disclose.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (PDF 231 KB)

Supplementary file2 (DOCX 14 KB)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Althans, A.R., Byrd, T., Suppok, R. et al. The role of holistic review in diversifying graduate medical education: a systematic review and meta-analysis. Global Surg Educ 3, 67 (2024). https://doi.org/10.1007/s44186-024-00269-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s44186-024-00269-y

Keywords

Search

Navigation