Standard Setting for Mastery Learning

Abstract

Medical education has gradually shifted toward a competency-based approach over the past decade. To comply with expectations of accrediting bodies, medical educators have been asked to demonstrate that graduating students and residents are competent to advance to the next phase of training. However, traditional clinical skills evaluations are subjective and often based on one-on-one interactions with supervising faculty members. Additional concerns have been raised about uneven clinical experiences leading to variable clinical skill acquisition among graduates. Mastery learning curricula allow medical schools and GME programs to document clinical skill acquisition with high reliability and make valid decisions about trainee competency. In this chapter, we discuss historical standard setting methods used in health professions education and set a path forward for fair, reasonable, and evidence-based standard setting in mastery learning environments.

Appendix 6.1: Standard Setting Packets for Traditional Angoff and Hofstee and Mastery Angoff and Patient-Safety Methods for Simulated Lumbar Puncture

Performance data (reviewing these data may be useful for traditional Angoff and Hofstee approaches )

This table shows sample pretest and posttest data from a pilot group of 57 internal medicine residents performing a simulated lumbar puncture procedure. Overall pretest and posttest means/standard deviations are displayed as well as the frequency of each overall score at pre- and posttest

% Correct	Pretest frequency	Posttest frequency
10%	1	0
19%	1	0
24%	6	0
29%	4	0
33%	5	0
38%	7	0
43%	6	0
48%	6	0
52%	3	1
57%	3	0
62%	4	0
67%	5	1
71%	1	0
76%	3	0
81%	1	1
86%	1	5
90%	0	8
95%	0	15
100%	0	26
	Mean = 46.3%	Mean = 94.4%
	SD = 17.6%	SD = 8.5%

• A. Traditional Angoff Method

  • 1. Select the judges.

  • 2. Discuss the purpose of the test, the curriculum and assessment, the nature of the examinees, and what constitutes adequate and inadequate skills/knowledge. Review baseline performance data.

  • 3. Define the “borderline” group, a group that has a 50–50 chance of passing.

  • 4. Read the first item.

  • 5. Each judge estimates the proportion of the borderline group that would perform it correctly.

  • 6. The ratings are recorded for all to see, discuss, and change as appropriate.

  • 7. Repeat steps 4–6 for each item.

  • 8. Calculate the passing score by averaging the estimates of all judges for each item and summing the items.

  • 9. Use the checklist below^a to do this exercise.

Checklist item	Pilot pretest data (%)	% of borderline residents who perform each step correctly
Clean the skin with betadine (may not use chlorhexidine) × 3	30
Drape the patient	91
Use 1% lidocaine to form a wheal at intended site	54
Numb deeper structure (larger needle)	54
Insert spinal needle advancing toward umbilicus (may be more cephalad depending on how flexed the spine)	65
Bevel must be in correct direction	46
Slowly advance the needle with periodic checking for CSF (removal of stylet) until enter space	23
Measure opening pressure	14

1. ^aThis is a partial checklist adapted from Barsuk et al. [26]. In an actual standard setting exercise, insert complete assessment tool with performance data

• B. Traditional Hofstee Method

  • 1. Select the judges.

  • 2. Discuss the purpose of the test, the curriculum and assessment, the nature of the examinees, and what constitutes adequate and inadequate skills/knowledge. Review baseline performance data.

  • 3. Review the test in detail.

  • 4. Ask the judges to answer four questions:

    • (a) What is the minimum acceptable required passing score?

    • (b) What is the maximum acceptable required passing score?

    • (c) What is the minimum acceptable fail rate?

    • (d) What is the maximum acceptable fail rate?

  • 5. After the test is given, graph the distribution of scores and select the cut score as described by De Gruiter^a

Clinical skill standard setting Hofstee method
	Minimum acceptable required passing score	Maximum acceptable required passing score	Minimum acceptable fail rate	Maximum acceptable fail rate
Clinical skill

1. ^aDe Gruiter [27]

• C. Mastery Angoff Method

• 1. Select the judges.

• 2. Discuss the purpose of the test, the curriculum and assessment, the nature of the examinees, and what constitutes adequate and inadequate skills/knowledge.

  • (a) Mastery learning: residents can continue to practice and retest until they achieve the passing standard (no penalty for taking a longer time or multiple retests).

  • (b) Past performance data is not relevant, since residents can keep practicing until they can accomplish even difficult items.

• 3. Define the “well prepared to succeed” group: the standard reflects the expected performance in the sim lab of residents who are:

  • (a) Well prepared to perform the procedure

  • (b) Safely and successfully

  • (c) On live patients

  • (d) With minimal supervision

• 4. Read the first item.

• 5. Each judge estimates the proportion of the “well prepared” group that would get it right (or the probability that any individual “well prepared” resident would get it right).

• 6. The ratings are recorded for all to see, discuss, and change as appropriate.

• 7. Repeat steps 4–6 for each item.

• 8. Calculate the passing score by averaging the estimates of all judges for each item and summing the items.

• 9. Use the checklist below^a to do this exercise.

Checklist item	% of well-prepared residents who accomplish this item correctly in the sim lab
Clean the skin with betadine (may not use chlorhexidine) × 3
Drape the patient
Use 1% lidocaine to form a wheal at intended site
Numb deeper structure (larger needle)
Insert spinal needle advancing toward umbilicus (may be more cephalad depending on how flexed the spine)
Bevel must be in correct direction
Slowly advance the needle with periodic checking for CSF (removal of stylet) until enter space
Measure opening pressure

1. ^aThis is a partial checklist adapted from Barsuk et al. [26]. In an actual standard setting exercise, insert complete assessment tool

• D. Patient-Safety Method

• 1. Select the judges.

• 2. Discuss the purpose of the test, the curriculum, assessment, and the nature of the examinees.

  • (a) Mastery learning: residents can continue to practice and retest until they achieve the passing standard (no penalty for taking a longer time or multiple retests).

• 3. Determine dimensions relevant to patient safety.

  In this case we will consider relevant dimensions to be

  • (a) Patient or provider safety

  • (b) Patient comfort

  • (c) The outcome of the procedure

• 4. For each item, each judge indicates whether performance or non-performance of this item would impact each of these dimensions.

• 5. Do this for the skills checklist below^a

• 6. Set standards separately for critical and non-critical items.

  • (a) An item that impacts any one of the three dimensions is considered a critical item.

  • (b) An item that does not impact any one of these dimensions is considered a non-critical item.

• 7. Average across judges to determine:

  • (a) Which items are critical or non-critical

  • (b) Passing scores for critical and non-critical items

• 8. Standards are not connected. Accomplishing non-critical items does not compensate for non-performance of critical items.

Checklist itema	Impacts safety?		Impacts comfort?		Impacts outcome?
Clean the skin with betadine (may not use chlorhexidine) × 3	Yes	No	Yes	No	Yes	No
Drape the patient	Yes	No	Yes	No	Yes	No
Use 1% lidocaine to form a wheal at intended site	Yes	No	Yes	No	Yes	No
Numb deeper structure (larger needle)	Yes	No	Yes	No	Yes	No
Insert spinal needle advancing toward umbilicus (may be more cephalad depending on how flexed the spine)	Yes	No	Yes	No	Yes	No
Bevel must be in correct direction	Yes	No	Yes	No	Yes	No
Slowly advance the needle with periodic checking for CSF (removal of stylet) until enter space	Yes	No	Yes	No	Yes	No
Measure opening pressure	Yes	No	Yes	No	Yes	No

1. ^aThis is a partial checklist adapted from Barsuk et al. [26]. In an actual standard setting exercise, insert complete assessment tool

Setting the Standard

The passing standard represents performance in the simulation lab, before performing the procedure on live patients. Residents can continue to practice and retest until they achieve the passing standard; there is no penalty for taking a longer time or multiple retests.

• 1. What should be the passing standard for critical items, i.e., items that impact patient or provider safety, patient comfort, or procedure outcome? What proportion of critical items should residents perform correctly in the sim lab before performing the procedure on live patients with minimal supervision?

  ______%

• 2. What should be the passing standard for non-critical items , i.e., items that do not impact patient or provider safety, patient comfort, or procedure outcome? What proportion of non-critical items should residents perform correctly in the sim lab before performing the procedure on live patients with minimal supervision?

  ______%

Please add any comments you may have about these standard setting procedures: