Item-specific and relational encoding are effective at reducing the illusion of competence

Abstract

Metamemory, or the ability to understand the capacities of one’s own memory, is important for learning. To investigate questions surrounding metamemory, researchers commonly have participants make judgments of learning (JOLs) at encoding, in which participants rate their likelihood of recalling the target in a cue–target word pair when shown only the cue at test. However, the associative direction of cue–target pairs can affect the calibration of JOLs. Unlike forward associates (e.g., credit–card), in which JOLs often accurately predict recall, an illusion of competence has been reported for backward associates (e.g., card–credit), symmetrical associates (e.g., salt–pepper), and unrelated cue–target pairs (e.g., artery–bronze) such that JOLs overestimate later recall. The present study evaluates whether the illusion of competence can be reduced when participants apply deep item-specific or relational encoding tasks relative to silent reading. Across two experiments, we show that both item-specific and relational encoding strategies reduce the illusion of competence for backward associates and unrelated pairs while improving the calibration between JOLs and recall. Our findings suggest that these encoding strategies are effective at reducing the illusion of competence, with increased calibration primarily reflecting improved recall. Thus, item-specific and relational encoding strategies primarily affect retrieval processes rather than metacognitive processes that participants engage in at encoding.

Appendices

Appendix

For both experiments, we assessed whether item-specific or relational encoding instructions affected the resolution between JOLs and recall. Relative accuracy or resolution refers to the degree to which a person’s JOL rating discriminates between what is and what is not remembered (Rhodes, 2016). Unlike calibration, which can be assessed through plots, resolution is commonly assessed via Goodman–Kruskal gamma correlations. The gamma coefficient represents a measure of association between − 1 and + 1, with resolution decreasing as gamma approaches zero. Positive values denote the degree that remembered items were given high JOLs and non-remembered items low JOLs, while negative gamma values denote the inverse of this pattern (Nelson, 1984). While the illusion of competence is generally assessed in terms of calibration (e.g., Koriat & Bjork, 2005), we note that item-specific and relational encoding strategies may additionally improve resolution, given that resolution is affected whenever an encoding task affords participants with an opportunity to adjust their JOLs (i.e., modifying JOLs based on previous trials). Thus, for completeness, we report a series of analyses assessing changes in resolution for each experiment.

Experiment 1: resolution

Following the procedure used by Nelson and colleagues (Dunlosky & Nelson, 1992, 1994; Nelson, 1984), we computed Goodman–Kruskal gamma correlations (G) between JOLs and recall for each participant for each of the four pair types (forward, backward, symmetrical, and unrelated; Table 5 reports mean Gs and 95% CIs as functions of pair type and encoding group). To test for changes in resolution, we assessed differences in mean G using a 3 (Encoding Group: Item-Specific vs. Relational vs Read) × 4 (Pair Type: Forward vs. Backward vs. Symmetrical vs. Unrelated) mixed ANOVA. Overall, main effects/interactions were only marginally significant, Fs ≥ 1.94; ps ≤ 0.07, p_BICs > 0.99; however, planned follow-up analyses were still carried out.

For forward pairs, both item-specific and relational encoding resulted in reduced resolution compared to silent reading (0.10 vs. 0.13 vs. 0.35, respectively). All comparisons differed significantly (ts ≥ 2.56, ds ≥ 0.64), except for the comparison between item-specific and relational encoding, t < 1, p = 0.97, p_BIC = 0.88. This pattern subsequently extended to backward pairs (0.12 vs. 0.07 vs. 0.24), though only the comparison between the relational encoding and read groups was significant, t(57) = 2.34, SEM = 0.07, d = 0.60, and all other comparisons for backward pairs were non-significant, ts ≤ 1.63, ps ≥ 0.11, p_BICs ≥ 0.67. For symmetrical pairs, G was again lower for item-specific and relational encoding relative to the read group (0.15 vs. 0.13 vs. 0.23), however, all comparisons failed to reach conventional significance, ts ≤ 1.53, ps ≥ 0.13, p_BICs ≥ 0.70. Finally, for unrelated pairs, resolution was increased for participants who completed item-specific (0.26) and relational encoding tasks (0.33) relative to participants in the read group (0.20). However, again, all comparisons failed to reach significance, ts ≤ 1.06, ps ≥ 0.29, p_BICs ≥ 0.81. Thus, while item-specific and relational encoding strategies are effective at reducing the illusion of competence, this reduction appears to occur primarily due to changes in calibration rather than resolution.

Experiment 2: resolution

Next, we assessed whether item-specific or relational encoding instructions influenced the resolution between JOLs and recall (see Table 6 for Mean Gs and 95% CIs for all comparisons). A 3 (Encoding Group: Item-Specific vs. Relational vs. Read) × 4 (Pair Type: Forward vs. Backward vs. Symmetrical vs. Unrelated) mixed ANOVA was used to test for differences in resolution as functions of encoding group and pair type. Overall, this analysis yielded a significant main effect of encoding group, F(2, 99) = 3.59, MSE = 0.24, η_p² = 0.07. Collapsed across pair types, resolution was greater for participants in the read group (0.19) relative to the item-specific (0.10) and relational encoding groups (0.03). All comparisons were non-significant, ts ≤ 1.04, ps ≥ 0.30, p_BICs ≥ 0.82, except for the comparison between the read and relational groups, t(66) = 3.01, SEM = 0.05, d = 0.74. Additionally, this analysis revealed a significant effect of pair type, F(3, 297) = 4.29, MSE = 0.19, η_p² = 0.04. Post hoc testing indicated that resolution was greatest for unrelated pairs (0.19), followed by symmetrical pairs (0.17), forward pairs (0.08), and backward pairs (0.01). Resolution for backward pairs was significantly lower relative to symmetrical and unrelated pairs, ts ≥ 3.22, ds ≥ 0.37, though comparison between all other pair types were non-significant, ts ≤ 1.49, ps ≥ 0.14, p_BICs ≥ 0.72. Additionally, the Encoding Group × Pair Type interaction was non-significant, F(6, 297) = 1.69, MSE = 0.19, p = 0.12, p_BIC > 0.99. Thus, like Experiment 1, item-specific and relational encoding reduced the illusion of competence primarily through improved calibration than resolution.

Cross-experimental analysis

Because participants in the item-specific and relational encoding groups in Experiment 2 were required to verbalize their encoding processes, it is possible that this procedure affected the magnitude of the JOLs and/or their recall performance. We tested this possibility using a 2 (Experiment) × 2 (Measure: JOL vs. Recall) × 3 (Encoding Group: Item-Specific vs. Relational vs. Read) × 4 (Pair Type: Forward vs. Backward vs. Symmetrical vs. Unrelated) mixed ANOVA. The only reliable interaction that emerged was the Experiment × Measure × Direction interaction, F(3, 552) = 3.94, MSE = 128.35, η_p² = 0.02. All other interactions with Experiment, including the four-way interaction, were non-significant, Fs ≤ 2.02 ps ≥ 0.06, p_BICs ≥ 0.64.

Overall, collapsed across encoding groups, mean JOL ratings did not differ between Experiments 1 and 2 for forward pairs (70.23 vs. 66.58, respectively), t(188) = 1.67, SEM = 2.23, p = 0.10, p_BIC = 0.77, or backward pairs (69.26 vs. 66.55), t(188) = 1.19, SEM = 2.29, p = 0.24 p_BIC = 0.87. For symmetrical pairs, JOLs in Experiment 1 were marginally greater than Experiment 2 (75.35 vs. 71.22), t(188) = 1.81, SEM = 2.32, p = 0.07 p_BIC = 0.73, while JOLs for unelated pairs were marginally lower in Experiment 1 relative to Experiment 2 (33.69 vs. 39.01), t(188) = 1.81, SEM = 2.94, p = 0.07 p_BIC = 0.72. Thus, across pair types, having participants engage in the think-aloud procedure in Experiment 2 did not affect their JOLs.

Regarding recall, no differences emerged between experiments for forward pairs (73.92 vs. 73.72), t < 1, SEM = 2.87, p = 0.92 p_BIC = 0.93, or symmetrical pairs (72.70 vs. 75.99), t(188) = 1.22, SEM = 2.64, p = 0.22 p_BIC = 0.87. However, for backward pairs, recall was greater in Experiment 2 than Experiment 1 for backward pairs (49.27 vs. 59.16), t(188) = 3.01, SEM = 3.33, d = 0.44, and unrelated pairs (20.91 vs. 28.64), t(188) = 2.27, SEM = 3.41, d = 0.33. Thus, the additional encoding afforded by the think-aloud task boosted recall, but only for more challenging backward and unrelated pairs. Importantly however, the item-specific and relational encoding effects produced similar effects on reducing the illusion of competence on both experiments, demonstrating that participants were indeed applying item-specific and relational processing tasks effectively in Experiment 1 when encoding was completed silently.

Additionally, we examined experiment differences in calibration plots and resolution. First, cross-experimental differences in calibration plots were assessed via a 2 (Experiment) × 3 (Encoding Group: Item-Specific vs. Relational vs. Read) × 4 (Pair Type: Forward vs. Backward vs. Symmetrical vs. Unrelated) × 11 (JOL Increment) mixed ANOVA. Overall, this analysis yielded a significant Experiment × Pair Type interaction, F(3, 546) = 12.57, MSE = 1640.37, η_p² = 0.12. However, all other interactions, including the four-way interaction, failed to reach significance, Fs ≤ 1.69, ps ≥ 0.08, p_BICs > 0.99. Regarding resolution, a 2 (Experiment) 3 × (Encoding Group: Item-Specific vs. Relational vs Read) × 4 (Pair Type: Forward vs. Backward vs. Symmetrical vs. Unrelated) mixed ANOVA confirmed that mean G did not differ as a function of experiment, as no interactions with Experiment were detected, Fs ≤ 1.72, ps ≥ 0.16, p_BICs > 0.99. Thus, changes in calibration and resolution across pair types/encoding groups did not differ between experiments.

Table 1 Mean associative strength summary statistics forward, backward, and symmetrical pairs

Table 2 Summary statistics for cue and target concreteness, length, and frequency item properties as a function of pair type

Table 3 Comparison of mean JOL ratings and correct recall percentages across pair directions for each encoding group in Experiment 1

Table 4 Comparison of mean JOL ratings and correct recall percentages across all pair directions for each encoding group in Experiment 2

Table 5 Mean (± 95% CI) Goodman–Kruskal gamma correlations between JOLs and recall for each encoding group as a function of pair type in Experiment 1

Table 6 Mean (± 95% CI) Goodman–Kruskal gamma correlations between JOLs and recall for each encoding group as a function of pair type in Experiment 2