Experiences and Lessons Learned Creating and Validating Concept Inventories for Cybersecurity

Abstract

We reflect on our ongoing journey in the educational Cybersecurity Assessment Tools (CATS) Project to create two concept inventories for cybersecurity. We identify key steps in this journey and important questions we faced. We explain the decisions we made and discuss the consequences of those decisions, highlighting what worked well and what might have gone better.

The CATS Project is creating and validating two concept inventories—conceptual tests of understanding—that can be used to measure the effectiveness of various approaches to teaching and learning cybersecurity. The Cybersecurity Concept Inventory (CCI) is for students who have recently completed any first course in cybersecurity; the Cybersecurity Curriculum Assessment (CCA) is for students who have recently completed an undergraduate major or track in cybersecurity. Each assessment tool comprises 25 multiple-choice questions (MCQs) of various difficulties that target the same five core concepts, but the CCA assumes greater technical background.

Key steps include defining project scope, identifying the core concepts, uncovering student misconceptions, creating scenarios, drafting question stems, develo** distractor answer choices, generating educational materials, performing expert reviews, recruiting student subjects, organizing workshops, building community acceptance, forming a team and nurturing collaboration, adopting tools, and obtaining and using funding.

Creating effective MCQs is difficult and time-consuming, and cybersecurity presents special challenges. Because cybersecurity issues are often subtle, where the adversarial model and details matter greatly, it is challenging to construct MCQs for which there is exactly one best but non-obvious answer. We hope that our experiences and lessons learned may help others create more effective concept inventories and assessments in STEM.

A A Crowdsourcing Experiment

To investigate how subjects respond to the initial and final versions of the CCA switchbox question (see Sects. 3.5–3.7), we polled 100 workers on Amazon Mechanical Turk (AMT)  [27]. We also explored the strategy of generating distractors through crowdsourcing, continuing a previous research idea of ours  [38, 39]. Specifically, we sought responses from another 200 workers to the initial and final stems without providing any alternatives.

On March 29–30, 2020, at separate times, we posted four separate tasks on AMT. Tasks 1 and 2 presented the CCA switchbox test item with alternatives, for the initial and final versions of the test item, respectively. Tasks 3 and 4 presented the CCA switchbox stem with no alternatives, for the initial and final versions of the stem, respectively.

For Tasks 1–2, we solicited 50 workers each, and for Tasks 3–4, we solicited 100 workers each. For each task we sought human workers who graduated from college with a major in computer science or related field. We offered a reward of $0.25 per completed valid response and set a time limit of 10 min per task.

Expecting computer bots and human cheaters (people who do not make a genuine effort to answer the question), we included two control questions (see Fig. 4), in addition to the main test item. Because we expected many humans to lie about their college major, we constructed the first control question to detect subjects who were not human or who knew nothing about computer science. All computer science majors should be very familiar with the binary number system. We expect that most bots will be unable to perform the visual, linguistic, and cognitive processing required to answer Question 1. Because $11\,+\,62\,=\,73\,=\,64\,+\,8\,+\,1$, the answer to Question 1 is 1001001.

We received a total of 547 responses, of which we deemed 425(78%) valid. As often happens with AMT, we received more responses than we had solicited, because some workers responded directly without payment to our SurveyMonkey form, bypassing AMT. More specifically, we received 40, 108, 194, 205 responses for Tasks 1–4, respectively, for which 40(100%), 108(100%), 120(62%), 157(77%) were valid, respectively. In this filtering, we considered a response valid if and only if it was non-blank and appeared to answer the question in a meaningful way. We excluded responses that did not pertain to the subject matter (e.g., song lyrics), or that did not appear to reflect genuine effort (e.g., “OPEN ENDED RESPONSE”).

Fig. 4.

Control questions for the crowdsourcing experiment on AMT. Question 1 aims to exclude bots and subjects who do not know any computer science.

Only ten (3%) of the valid responses included a correct answer to Question 1. Only 27(7%) of the workers with valid responses stated that they majored in computer science or related field, of whom only two (7%) answered Question 1 correctly. Thus, the workers who responded to our tasks are very different from the intended population for the CCA.

Originally, we had planned to deem a response valid if and only if it included answers to all three questions, with correct answers to each of the two control questions. Instead, we continued to analyze our data adopting the extremely lenient definition described above, understanding that the results would not be relevant to the CCA.

Fig. 5.

Histogram of responses to the CCA switchbox test item, from AMT workers, for the initial and final versions of the test item. There were 148 valid responses total, 40 for the initial version, and 108 for the final version.

We processed results from Tasks 3–4 as follows, separately for each task. First, we discarded the invalid responses. Second, we identified which valid responses matched existing alternatives. Third, we grouped the remaining (non-matching) valid new responses into equivalency classes. Fourth, we refined a canonical response for each of these equivalency classes (see Fig. 8). The most time-consuming steps were filtering the responses for validity and refining the canonical responses.

Especially considering that the AMT worker population differs from the target CCA audience, it is important to recognize that the best distractors for the CCA are not necessarily the most popular responses from AMT workers. For this reason, we examined all responses. Figure 8 includes two examples, $t*$ for initial version (0.7%) and $t*$ for final version (1.2%), of interesting unpopular responses. Also, the alternatives should satisfy various constraints, including being non-overlap** (see Sect. 3.7), so one should not simply automatically choose the four most popular distractors.

Figures 5, 6, 7 and 8 summarize our findings. Figure 5 does not reveal any dramatic change in responses between the initial and final versions of the test item. It is striking how strongly the workers prefer Distractor E over the correct choice A. Perhaps the workers, who do not know network security, find Distractor E the most understandable and logical choice. Also, its broad wording may be appealing. In Sect. 3.7, we explain how we reworded Distractor E to ensure that Alternative A is now unarguably better than E.

Fig. 6.

Histogram for equivalency classes of selected open-ended responses generated from the CCA switchbox stem. These responses are from AMT workers, for the initial and final versions of the stem, presented without any alternatives. There were 120 valid responses for the initial version, and 157 for the final version. A–E are the original alternatives, and t1–t4 are the four most frequent new responses generated by the workers. These responses include two alternate phrasings of the correct answer: t2 for the initial version, and t1 for the final version. Percents are with respect to all valid responses and hence do not add up to 100%.

Fig. 7.

Histogram for equivalency classes of selected open-ended responses generated from the CCA switchbox stem. These responses are from AMT workers, for the initial and final versions of the stem, presented without any alternatives. There were 120 valid responses for the initial version, and 157 for the final version. A–E are the original alternatives, and t1–t5 are the four most frequent new distractors generated by the workers. The alternate correct responses t2 (initial version) and t1 (final version) are grouped with Alternative A. Percents are with respect to all valid responses and hence do not add up to 100%.

Figure 6 shows the popularity of worker-generated responses to the stem, when workers were not given any alternatives. After making Fig. 6, we realized it contains a mistake: responses t2 (initial version) and t1 (final version) are correct answers, which should have been matched and grouped with Alternative A. We nevertheless include this figure because it is informative. Especially for the initial version of the stem, it is notable how more popular the two worker-generated distractors t2 (initial version) and t1 (final version) are than our distractors. This finding is not unexpected, because, in the final version, we had intentionally chosen Alternative A over t1, to make the correct answer less obvious (see Sect. 3.7). These data support our belief that subjects would find t1 more popular than Alternative A.

Fig. 8.

Existing alternatives (A–E), and refinements of the five most frequent new responses (t1–t5) generated for the CCA switchbox stem, from AMT workers, for the initial and final versions of the stem. There were 120 valid responses for the initial version, and 157 for the final version. Distractors $t*$ for initial version (0.7%) and $t*$ for final version (1.2%) are examples of interesting unpopular responses. Percents are with respect to all valid responses.

Figure 7 is the corrected version of Fig. 6, with responses t2 (initial version) and t1 (final version) grouped with the correct answer A. Although new distractor t1 (initial version) was very popular, its broad nebulous form makes it unlikely to contribute useful information about student conceptual understanding. For the first version of the stem, we identified seven equivalency classes of new distractors (excluding the new alternate correct answers); for the final version we identified 12.

Because the population of these workers differs greatly from the intended audience for the CCA, these results should not be used to make any inferences about the switchbox test item for the CCA’s intended audience. Nevertheless, our experiment illustrates some notable facts about crowdsourcing with AMT. (1) Collecting data is fast and inexpensive: we collected all of our responses within 24 hours paying a total of less than $40 in worker fees (we did not pay for any invalid responses). (2) We had virtually no control over the selection of workers, and almost all of them seemed ill-suited for the task (did not answer Question 1 correctly). Nevertheless, several of the open-ended responses reflected a thoughtful understanding of the scenario. (3) Tasks 3–4 did produce distractors (new and old) of note for the worker population, thereby illustrating the potential of using crowdsourcing to generate distractors, if the selected population could be adequately controlled. (4) Even when the worker population differs from the desired target population, their responses can be useful if they inspire test developers to improve test items and to think of effective distractors.

Despite our disappointment with workers answering Question 1 incorrectly, the experience helped us refine the switchbox test item. Reflecting on the data, the problem development team met and made some improvements to the scenario and distractors (see discussions near the ends of Sects. 3.5 and 3.7). Even though the AMT workers represent a different population than our CCA target audience, their responses helped direct our attention to potential ways to improve the test item.

We strongly believe in the potential of using crowdsourcing to help generate distractors and improve test items. Being able to verify the credentials of workers assuredly (e.g., cryptographically) would greatly enhance the value of AMT.