1 Introduction

This paper investigates how learning happens during mainstream, time-bounded collaborative events such as hackathons, to what extent learning happens, and how we can facilitate it. With the rising popularity of hackathons from niche events to mainstream collaboration platforms (Taylor & Clarke, 2018) also came an increase in research on hackathons and hackathon-like events (Falk Olesen & Halskov, 2020). Such events are often aimed at the development of digital technologies (Pe-Than et al., 2019) while participants commonly use a variety of digital technologies to carry out their projects, communicate with peers, and organize their work (Mendes et al., 2022). In such settings, and according to related research, learning is among the most prominent reasons and motivations for organizing hackathons or participating in them (Pe-Than et al., 2019; Falk Olesen & Halskov, 2020). For example, participants join hackathons to learn something or to practice some skill, and organizers state that they aim to facilitate or foster learning. Additionally, hackathons – or similar events – have found their way into classrooms and lecture halls as a means to promote collaboration, to facilitate creativity and idea generation, or to explore new approaches to learning (Nandi & Mandernach, 2016; Pe-Than et al., 2019; Falk Olesen & Halskov, 2020). Likewise, hackathons are proposed to fill the gap in e-learning settings to foster interaction and collaboration between otherwise isolated students Giray (2021).

Similarities to long-standing instructional approaches and learning paradigms, for example, Project-based Learning PBL and problem-based learning, indeed support the idea that hackathon-like events can facilitate and foster learning (La Place et al., 2017; Horton et al., 2018; Nolte et al., 2020). For example, Chounta et al. (2017) argued that events such as hackathons and dev camps can be used as a blueprint for re-inventing PBL in Higher Education and to foster the acquisition of soft skills such as problem-solving, collaboration, and communication. At the same time, introducing hackathons in formal education settings may lead to the same issues as PBL: a) constraints regarding the systematization and standardization of knowledge, b) limitations regarding the assessment of learning outcomes, and c) challenges for the new roles and responsibilities of teachers and instructors (Thomas, 2000).

1.1 Research questions

In this work, we study the state of the art concerning the research of learning in hackathon-like events from the perspectives of standardization and systematization of knowledge and the evaluation of learning outcomes. Then, we explore the opportunities and challenges that hackathons may introduce for formal education and focus on the new roles and responsibilities for teachers and instructors.

Our research questions aim to provide insights regarding (a) evidence for the effectiveness of hackathons as a learning approach, (b) how planning, implementation and management of hackathons that aim to learning are practiced, and how they, in turn, relate to learning outcomes, and (c) existing practices in hackathons that point towards the systematization and standardization of knowledge and learning outcomes. To that end, we have formulated three research questions:

  1. RQ 1

    How do hackathon-like events promote learning? We want to document the processes, tactics and design elements, such as the existence of a theme and related learning goal, that hackathons and other hackathon-like events employ to promote learning.

  2. RQ 2

    How are learning activities orchestrated in the context of hackathon-like events? Our goal is to gain insight regarding the learning context and learning designs in which such events occur; for example, whether informal learning contexts are more appropriate than formal education or if learning designs that introduce interventions are commonplace.

  3. RQ 3

    How are knowledge and learning outcomes standardized and systematized in the context of hackathon-like events? We aim to retrieve and collect information regarding systematization and standardization processes that can enhance learning in hackathons and support integration of hackathon events in learning contexts at scale; for example, shared strategies for promoting reproducibility and reflection, or learning designs that allow for interventions and objective assessment of outcomes.

To answer the research questions, we conducted a Systematic Literature Review (SLR) following the Kitchenham guidelines (Kitchenham, 2004) and cross-validating using the PRISMA statement (Page et al., 2021). We argue that for learning to happen, one should purposefully design and prepare the conditions and context. The contribution of this work is two-fold:

  1. 1.

    to map the current landscape regarding learning in hackathons from the perspective of systematization, standardization, and evaluation of knowledge and learning;

  2. 2.

    to propose research directions toward rigorous learning designs that, on the one hand, deliberately capitalize learning and on the other hand, take advantage of hackathon-like events’ nature to promote and practice soft skills such as problem-solving and critical thinking that are necessary for the digital era (Chounta et al., 2023).

In the following sections, we provide an overview of related research on hackathons, and learning in hackathons. Then, we present our methodological approach and discuss our findings. We conclude with a contextualized discussion that is guided by our research questions and elaborates on the theoretical and practical implications of this work.

2 Background

2.1 Hackathons

The term hackathon is an amalgamation of the words “hacking" and “marathon". It describes time-bounded events that span from a couple of hours to multiple days. These events are at their core collaborative events (Pe-Than et al., 2019; Falk Olesen & Halskov, 2020) where teams collaborate on a project about a specific – and sometimes, self-chosen – topic. These projects may include ideation, design, prototy**, and/or development and are usually characterized by intensive use of technology (Pe-Than et al., 2019).

There are numerous names for similar event types in the literature, for example, makeathon (MacDowell et al., 2017), dev camp (Chounta et al., 2017), hack day, hack week, hackfest, codefest and codeathon. The different events may also take place in very different contexts. For example, at educational institutions, we can find events that are part of a course (Gama et al., 2021), events that are outside of, but related to, the curriculum (Fornós et al., 2022), or informal (student-led events) for fun, networking and/or for research (Nandi & Mandernach, 2016). Outside of educational institutions, organizations or companies lead similar events for the training of employees (Backert et al., 2022) or to develop new ideas (Nolte et al., 2018). We also find events organized by educational institutions in collaboration with organizations (Maaravi, 2020; Page et al., 2016) often used for mentoring for the students by professionals but also for organizational innovation as highlighted by Maaravi (2020).

Depending on the event, participants can be strangers or know each other before the event; they can come as a team or find one at the start of the event (Pe-Than et al., 2019, 2022).

In their literature review about hackathon research, Falk Olesen & Halskov (2020) differentiate between research with hackathons and research on hackathons. The former refers to research in which the hackathon is used as a tool or example for the larger research, the latter focuses on the hackathon itself.

An example of research with hackathons is the paper by Kazemitabar et al. (2023). Based on the positive effects of socially-shared emotion regulation (SSER) on group work and learning in groups Kazemitabar et al. (2023) investigated the relationship between SSER and shared mental models in a hackathon, as well as SSER and mutual trust. Among other things, they report significant correlations for two of the shared mental model factors ("task and communication skills" and "team dynamics and interaction") and SSER situation modification.

2.2 Learning in hackathons

Learning is often cited as a motivation for hackathon participants to join a hackathon (Gama et al., 2018; Falk Olesen & Halskov, 2020) and also for organizers, that is setting up a hackathon for participants to learn something (Karlsen & Løvlie, 2017). Hardin (2021) proposed that hackathons might offer a safe platform to learn through failure. They enable participants to take risks and try new things in an environment where failing reaches no further than not winning the hackathon prizes. This contrasts with student projects that may lead to bad grades in case of failure or work projects where demotion or dismissal might be the consequence of failing. Nandi & Mandernach (2016) looked into hackathons as informal learning platforms. Through qualitative research, they got positive feedback regarding learning from participants and showed that hackathon participants consistently demonstrated higher Grade Point Average (GPA) than non-participants.

Gama et al. (2018) aimed to use hackathons in the formal learning process in their Internet of Things course. The authors collected qualitative and quantitative data in their field study and found that students highlighted the learning aspect in their feedback, stating that they had learned more in the practical sessions – including the hackathon – than they did in the theoretical classes. Porras et al. (2019) conducted a literature review for investigating code camps and hackathons in education. Specifically, the authors aimed to identify the definition of these events, the educational structures they are used in, the skills and competencies that are emphasized in that context, and how these events have been used in Software Engineering and Computer Science education. They found that links between the reviewed studies and educational activities were scarce but still maintained the position that hackathons can play a part in education.

A recently published literature review by Oyetade et al. (2022) looked into hackathons in education with a specific focus on those explicitly implemented in an educational context. They investigated what research is available on educational hackathons and what advantages hackathons provide, looking into research from the past five years. They found that research on the benefits of hackathons in education is gaining traction, and they encourage the promotion of hackathons for teen audiences to foster collaboration and networking.

We argue that hackathon-like events can promote topic relevant content and skills as well as soft skills, such as collaboration, critical thinking, and problem-solving. Evidently, hackathons build on the same core ideas as other established, instructional approaches, such as PBL: teamwork on real-life challenges using tools and skills from the real world with support from more experienced mentors or coaches (Gama et al., 2018; Horton et al., 2018; Kopec et al., 2021). In the literature review, we present in this paper, we look into the potential of hackathon-like events to foster and promote learning. We investigate the learning goals for such events, the event structures to support these goals, and the methods and procedures employed in hackathons to assess learning outcomes.

3 Method

For our research purposes, we conducted a Systematic Literature Review (SLR) that followed the methodological guidelines described by Kitchenham (2004), following the example of other literature reviews on hackathon events (Medina Angarita & Nolte, 2020; Porras et al., 2019), hackathons in education (Porras et al., 2019) and learning (Topali et al., 2023). Kitchenham (2004) proposed a guideline for systematic literature reviews directed at researchers in the software engineering field based on guidelines directed at medical researchers and adapted to target software engineering more directly. The guideline divides the review process into 3 phases: planning, conducting, and reporting the review. The planning phase is further broken down into identifying the need for a review and develo** a protocol. Conducting the review is divided into identifying the research, selecting primary studies, assessing the study quality, extracting and monitoring data, and synthesizing data. The last phase of reporting is not further broken down. The Kitchenham (2004) guidelines overlap with the PRISMA statement (Page et al., 2021) which was used to certify every step of the process.

Table 1 Search terms
Full size table

3.1 The search terms, inclusion and exclusion criteria

We determined our search terms based on the research questions and related work. We split the search terms into two groups: (1) the learning aspect, for which we wanted to account for formal as well as informal settings, include self-initiated learning efforts and not rule out incidental or collateral learning (Dewey, 1953) and (2) the event type. Relevant literature reviews on hackathons used the event types themselves as search terms; for example, Porras et al. (2019) used the terms "Hackathon" and "Code camp", Medina Angarita & Nolte (2020) used the terms “hackathon”, “codefest” and “coding competition”, and lastly Oyetade et al. (2022) and Chau & Gerber (2023) used the term "Hackathon" only. For our work, we used 13 terms to describe relevant event types with the aim to be inclusive: hackathon, devcamp, makeathon, hack day, dev camp, datathon, hackfest, code fest, codefest, hack fest, hack week, codeathon, and data fest.

In learning and education-related reviews, we found the usage of "education" (Kurniawan et al., 2019; Porras et al., 2019; Oyetade et al., 2022), “learning” (Kurniawan et al., 2019; Oyetade et al., 2022) and "training" (Kurniawan et al., 2019). We complemented this with additional terms – that is, studying, tutoring, and teaching – to overall account for active and intentional learning, self-initiated and externally-initiated learning. Table 1 shows the final search terms.

To identify relevant papers for our research, we defined the following inclusion and exclusion criteria:

  • Exclusion: We excluded papers from the review that met at least one of the following criteria:

    • Papers in languages other than English;

    • Papers shorter than 6 pages;

    • Abstracts and extended abstracts;

    • Papers that were not peer-reviewed;

    • Secondary and tertiary studies;

  • Inclusion: From the remaining papers, we searched for papers that met the following criteria:

    • They explored aspects of (collateral/incidental/intentional/etc.) learning;

    • They presented or focused on a hackathon-like event;

    • They investigated learning in the context of hackathon-like time-bounded events

3.2 Conducting the search

The search terms (Table 1) were used to collect papers from five digital databases within a 10-year span from 2012 to 2022. The digital databases we used were: the ACM Digital LibraryFootnote 1, IEEE XploreFootnote 2, ScienceDirectFootnote 3, ScopusFootnote 4 and Web of ScienceFootnote 5, as these were considered to encompass the relevant publications in Technology and Learning. These digital libraries were used in similar reviews on learning (for example, Topali et al. (2023)) and hackathons (Porras et al., 2019; Medina Angarita & Nolte, 2020; Oyetade et al., 2022). Before downloading the results from each database, we used the filtering mechanics provided by the database interface to limit the results according to our exclusion and inclusion criteria: papers written in English and published in peer reviewed journals or conference proceedings. After downloading the results, we joined them, checked for paper length, and removed duplicates. Figure 1 shows the number of papers for every phase of the process.

3.3 Filtering publications

We filtered the remaining papers in three rounds using the inclusion criteria. In particular, we rated each paper on whether there were claims or investigations regarding learning, if a hackathon-like event was the paper’s focus, and if the paper investigated learning in hackathon-like events. Thus, we excluded papers that only investigated either hackathons or learning (but not in combination) and papers that investigated learning (for example, in a course) and hackathon-like events (for example, as a platform to apply the already learned content) but did not investigate learning in the hackathon-like event. In the first screening, we only looked at the papers’ titles, abstracts, and keywords. For the second screening, we focused on each paper’s introduction and conclusion sections. In the third screening, we read the papers and also coded for these papers while filtering. The resulting amount of remaining papers after each round is shown in Fig. 1.

3.4 Coding process

The coding was performed by the authors of this paper. It took place in parallel to the third screening and in an iterative manner. We set up an initial coding scheme to code and filter the 96 papers left after the second screening. We used the first 18 papers (out of the 96) to discuss, establish a common ground among the reviewers, and fine-tune the coding scheme. Then, we coded the next 20 papers to establish that both reviewers had a common understanding and reached a consensus regarding the coding scheme. Having done so, we split the following 48 papers evenly and coded 24 papers each. Finally, we both read and coded the last 10 papers together. During the process, we had regular check-ins discussing the assigned codes and decisions.

Fig. 1
figure 1

Overview on the process for filtering and screening in this literature review. Figure adapted from Topali et al. (2023)

Full size image
Table 2 The coding scheme used to analyse the final papers
Full size table

3.5 Coding scheme

The coding scheme (Table 2) resulted from the iterative process described in Section 3.4. For some of the codes (such as [author_location], [organizer], [event_location], [event_count], [event_topic], [audience], [learning_goal], [interventions] and [learning_assessment]), we followed a bottom-up approach: first, we recorded detailed information and categorized this information afterward. This is indicated in the last column of Table 2. The map** is described in Section 3.6.1. The only code we neither used predefined options nor did a map** on was the [event_type]. This was because we wanted to collect what names (classifiers) the event organizers gave their own events.

Codes about the papers. For the [paper_method], we coded the work presented in the respective papers as application or conceptual. The location of the first author ([author_location]) was determined by the affiliation of the first author. To document the [intention] to investigate learning and [follow_through] we used binary codes (could both either be yes or no).

Codes about the reported hackathon-like events. The [organizer] of the hackathon-like event was noted from the papers to be categorized later following a bottom-up approach. To get an overview of the different event types, we did not pre-define options for [event_type]. The same was true for the [event_count]. As for the author location, we did not pre-define values for the [event_location]. The [event_topic] and target [audience] were also extracted as is from the papers to be categorized bottom-up subsequently. The [event_mode] was coded as either online, face to face, hybrid or unclear, in case the mode was not explicitly stated. Regarding [formality], this could be formal, semi-formal, informal or unclear. Formality here refers to formal (or informal) learning. A formal event would be part of the curriculum of a course, a semi-formal event could be attached to a course, but not mandatory and not part of the formal curriculum, and an informal event exists independently from curricula.

Codes related to learning and learning assessment. Similarly to the topic of the event, we extracted the [learning_goal] directly from the papers for this we focus on the main learning goal expressed in the paper. We had some ideas of possible [interventions] (such as mentoring, checkpoints and instruction), so we looked for those specifically but also kept an open mind to document additional interventions we might find and to then consequently group and categorize all interventions. Exploring what sort of [learning_assessment(s)] methods and tools are used in the context of hackathon-like events was a major part of our research goal. Therefore, we did not pre-define options for these and instead looked for what assessment types were used. For the reported [results], we originally intended to collect quantifiable data but ended up with the options positive results, non-significant results and no reported results.

3.6 Analysis

Figure 2 shows the map** between our final codes and the research questions, and some additional codes that refer to the overall description of the papers.

Fig. 2
figure 2

Overview of the map** of codes (blue/oval) and code-combinations (yellow/rectangle) to research questions

Full size image

3.6.1 Categorization of codes

As described in Section 3.5, we used a bottom-up approach for some of the codes. The map** of these values to categories is described as follows:

  • Codes about the papers. We grouped the determined [author_location(s)] in the associated continents.

  • Codes about the reported hackathon-like events. The [event_location(s)] were likewise grouped into continents. For the event [organizer(s)] we chose to group them into Higher Education Institution (HEI), HEI and others, organization(s) and unclear. The [event_count] was grouped into single event, multiple events within one course, multiple events in one repeated course, multiple (unrelated) events and event number unclear. We mapped [event_topic] to scientific disciplines according to the Deutsche Forschungsgemeinschaft DFG as presented in Güdler (2016). The target [audience] was grouped into HEI students, HEI students and young professionals, school students, any participants and unclear. Without explicitly mentioning schools, it was assumed that the term "students" referred to HEI students.

  • Codes related to learning and learning assessment. The [learning_goal] was – like the event topic – mapped to scientific disciplines. Here ’General Skills’ means that the hackathon event mainly targeted skills beyond topic knowledge which encompasses skills such as 21st century skills (Chounta et al., 2017), practice skills, promote critical thinking, support student engagement etc. Rennick et al. (2018), practice sustainable work (Islind & Norström, 2020), learn design thinking (Artiles & LeVine, 2015), cooperation and networking (Nandi & Mandernach, 2016). Soft skills such as networking, critical thinking and cooperation are thus included in the analysis under the General Skills category. We grouped the [interventions] in the categories feedback, framework to work in – this includes the checkpoints set up in some events –, knowledge dissemination – like seminars, workshops, talks or instructions – and mentoring (or coaching/tutoring). For the [learning_assessment], the codes are organized into four categories: Qualitative (such as Observations), Quantitative (for example, knowledge tests), Perceived Learning (such as self-reports), and Mixed Methods (like Grades). Due to the lack of description, it is unclear how grades are calculated, so those fall under Mixed Methods. For example, in Bonilla et al. (2020), they used course grades for the evaluation; however, it is unclear how much of the course content the hackathon covered and what topics were covered in the course exam. In Förster et al. (2021) likewise, a course grade is used; there, we know that the grade is calculated as \(10\%\) home assignments, \(40\%\) hackathons, and \(50\%\) final exam, but the authors only ever look at combined grades of assignments/hackathon or total grade.

3.6.2 Analysis of the promotion of learning in hackathon-like events (RQ 1)

To investigate whether the hackathon-like events in the papers under review promoted learning – and if so, how they did it – (RQ 1, see also Fig. 2), we examined the learning goals [learning_goal] and the employed assessment of learning [learning_assessment] for the described events. We considered the learning goal in relation to the general event topic [event_topic]. In regards to the assessment of learning, we considered these in the context of the categories described in the previous section: qualitative, perceived learning, quantitative and mixed methods. Lastly, we looked at how the assessment was done in relation to the events having – or not having – an explicit learning goal.

3.6.3 Analysis of the orchestration of learning in hackathon-like events (RQ 2)

To investigate how learning was orchestrated in hackathon-like events (RQ 2, see also Fig. 2), we first considered the event setup, i.e., what was the event mode [event_mode], the target audience [audience] and the learning formality [formality] in the event. We analyzed each of these codes individually, and combined to see possible correlations. Additionally, we looked at the interventions [interventions] implemented in the events that may foster learning, we considered how many papers report each type of intervention being used and what combinations occur. Lastly, we examined the combination of learning goals [learning_goal] and interventions to find potential correlations. The aim was to find a relation between what the event aimed to do and how the event was set up or to find common patterns in the setup of events.

3.6.4 Analysis of the standardization and systematization of learning outcomes in hackathon-like events (RQ 3)

For RQ 3 (see also Fig. 2) – the standardization and systematization of learning outcomes in hackathon-like events – we explored the combination of learning goals [learning_goal] and assessment of learning [learning_assessment] during hackathon-like events. We further investigated the reported learning results [results] and the combination of the learning goal, intervention [interventions], and assessment of learning, wherein we reduced the former two to binary values (has vs. has not). This was done to see if there are common patterns in the understanding and assessment of learning that could indicate an underlying standardized framework.

4 Results

It is important to note that multiple papers report on more than one hackathon-like event. Thus, the numbers reported here refer to the number of papers rather than the number of events. Namely, 23 papers report on a single event, while the other 16 report on two or more events. Of those 16, 3 have multiple events within the same course – one of which happened twice – and in 4 papers, the number of events is not specified.

4.1 Papers’ overview

To get an overview of the papers and the reported hackathon-like events, we looked at the following: the method of the paper [paper_method], the intention to investigate learning [intention] and the consequent follow through [follow_through], the event topic [event_topic], location of event [event_location] and author [author_location], the event organizers [organizer], event type [event_type] and the number of events [event_count] reported in the paper.

For the method of the papers [paper_method], we found that all 39 papers included a study with minor distinctions: 34 were application papers (that is, papers that focused on studying one or more hackathon-like events), 3 interviewed and/or surveyed participants from past events and 2 focused on a concept with proof-of-concept studies attached (see Table 3). Regarding the intentions of the authors to investigate learning [intention], 31 papers stated the intention to investigate learning and followed this up with an actual investigation looking into learning such as observation or querying participants. In 3 papers, we found the same aim but no appropriate follow-up activity. Another 3 did not explicitly claim to investigate learning but did so in their research. The remaining 2 neither planned to nor explicitly investigated learning but they provided valuable insights about learning in hackathon-like events relevant to this review.

Table 3 Overview on the research methods of the papers [paper_method]
Full size table
Fig. 3
figure 3

Map** of the locations of first authors (left) and events (right)

Full size image

Concerning the event location [event_location], 15 papers referenced events that took place in North America, 12 in Europe, 7 in South America, and 1 paper’s event took place in Asia. One paper presented an international array of locations (i.e., multiple events that take place on more than 1 continent). For the remaining 3 papers, the location of events was unclear. The events’ locations coincided to a large extent with the authors’ locations [author_location]. 16 of the first authors had affiliations in North America, 14 in Europe, 7 in South America, 1 first author’s affiliation was in Asia, and 1 first author had a dual affiliation (Europe and North America). Figure 3 presents the relationship between first authors’ locations and events’ locations. This shows that, for the most part, the first authors’ affiliation relates directly to the event’s location with few deviations.

While sometimes giving their events specific names [event_type], most papers (28) classify their events as "hackathons". 3 papers referred to their events as "makeathons" while the remaining 8 papers’ events were given various other names (see Fig. 4).

Next, we mapped the event topics to scientific disciplines [event_topic] (see Section 3.6.1) following the classification of DFG. 19 papers studied events that tackled topics from Engineering Science, 6 papers explored events that focused on topics from Humanities and Social Sciences, the events of 5 papers looked into Natural Sciences and for the remaining 5 papers the event topics were not explicitly mentioned (unclear).

The events described in 23 papers were organized [organizer] by one or more HEIs, for 7 papers HEIs and others organized the events, 5 papers reported on events organized by organizations, and for the remaining 4 papers the organizers were not explicitly stated (unclear).

Fig. 4
figure 4

Word cloud of event types [event_type] ascribed to the events in the papers

Full size image

4.2 Promotion of learning in hackathon-like events (RQ 1)

For RQ 1, we explored how the relevant literature on hackathon-like events addresses learning goals [learning_goal] and assessment of learning [learning_assessment]. Figure 5 depicts the relation between event topics [event_topic] and learning goals (see Table 6 in Appendix A for more details) and highlights the emphasis on Engineering Science, as well as the topic related learning goals (i.e. Engineering Science) within those events.

Fig. 5
figure 5

Map** of event topics (on the left) and learning goals (on the right)

Full size image

Overall we found 20 papers using perceived learning as an assessment, 11 used mixed methods, 5 did not assess learning, 2 used quantitative assessments and 1 used qualitative assessments. A detailed overview of the learning assessments can be found in Table 4.

Table 4 Overview on the assessment of learning found in the reviewed papers
Full size table

Figure 6 shows a comparison between papers reporting on events with explicit learning goals and papers that do not report explicit leaning goals in regards to the assessment of learning. Of the 12 papers referencing events with no explicit learning goal, 7 used perceived learning to evaluate learning, 4 used mixed methods and 1 did not measure learning. For the papers that named explicit learning goals, 13 used perceived learning, 7 mixed measures, 4 did not measure learning, 2 used quantitative, and 1 used qualitative measures.

Fig. 6
figure 6

Map** of explicit learning goals to method of assessment of learning

Full size image
Fig. 7
figure 7

Map** of the event mode, target audience and learning formality

Full size image

The findings suggest that perceived learning is the most used assessment method followed by mixed method approaches regardless of setting explicit learning goals or not. In other words, the definition of learning goals does not necessarily promote the systematic or standardized assessment of learning.

4.3 Orchestration of learning in hackathon-like events (RQ 2)

In order to assess the orchestration of learning in hackathon-like events, we looked at the event mode [event_mode] (in terms of mode of participation), the target audience [audience], the formality of the event regarding learning [formality], and the interventions [interventions]. 31 of the 39 papers reported on face-to-face (in-person) events, 3 on online events, and 1 on hybrid. For the remaining 4, the event mode was not clearly communicated in the paper (unclear). In 31 papers, the target audience for the reported event(s) were HEI students. For 3 papers the event(s) were open to any participant, 2 were targeting school students and for 1 event the target audience was HEI students and young professionals. For the remaining 2 events the target audience was not explicitly defined in the respective papers (unclear). In terms of the formality of the event – where formal means that the event is part of the curriculum – 11 papers reported on formal events, 6 on semi-formal – events that relate to some course but are optional – and 20 papers reported on informal events. For 2 papers, the formality of the events is unclear. Figure 7 presents an overview of the cross-section for event mode, target audience, and formality (see Table 7 in Appendix A for more details). We can see that most events targeted HEI students for all event modes. At the same time, all formal learning events also targeted HEI students.

Our analysis of related literature revealed various combinations of interventions to support learning (as defined in Section 3.6.1). All in all, we found interventions in 25 papers. Mentoring was mentioned in 21 papers, feedback in 8, knowledge dissemination in 5 and frameworks in 4. 9 papers mentioned only a single intervention type each: 6 papers mentioned only mentoring, in 2 they used knowledge dissemination and in 1 feedback. Further, 14 papers described combinations of two types of interventions, most of which include mentoring: 6 papers with mentoring and knowledge dissemination, 4 papers discussed mentoring and feedback, 3 discussed mentoring and frameworks and 1 focused on knowledge dissemination and feedback. Additionally, we found 1 paper using mentoring, feedback and knowledge dissemination, as well as 1 paper where all four types were used. Figure 8 shows the interventions per learning goal and paper with the respective event modes. One interesting finding is that 9 papers name learning goals for their events but do not employ any interventions to support learning during the event. Additionally, no noticeable relationship between the type of intervention and the event mode could be found.

Lastly, in Fig. 9, we see the connection between the events’ learning formality and the method used to assess learning outcomes. Most papers dealing with informal events assessed perceived learning, while for formal events mixed method assessments were extensively used. The 2 papers using only quantitative methods for assessing learning outcomes deal with informal events.

Fig. 8
figure 8

Combinations of learning goal, event mode and applied interventions per paper in the review

Full size image
Fig. 9
figure 9

Map** of the learning formality of the event and the type of learning assessment employed

Full size image

4.4 Standardization and Systematizing of learning outcomes in hackathon-like events (RQ 3)

For RQ 3, we explored how learning was assessed in the papers and how the assessment type related to the learning goal (as described in Section 4.2), the applied interventions (see Section 4.3) and the reported results. Figure 10 displays the matching of learning goals in the papers with the mapped learning assessments. The leading type of assessment for each learning goal is perceived learning followed by – or in the case of general skills on par with – mixed methods. This is not surprising, considering that these are the predominant assessment types found in the overall paper collection (see Table 8 in Appendix A for more details).

Next, we looked at the reported results (see Section 3.6.1). The paper that employed qualitative measures reported positive results, as do 16 of the papers measuring perceived learning. The remaining 4 papers that used perceived learning to assess outcomes did not report clear results. 8 of the papers using mixed methods reported positive results, 1 paper reported non-significant results, and 2 papers reported no clear results. The 2 papers that employed quantitative measures to assess learning reported positive results. The remaining 5 papers did not measure learning in hackathon-like events, and consequently, they did not report results.

Table 5 shows the combination of the learning goal, intervention, and assessment of learning. The largest group of papers (\(n=9\)) had an expressed learning goal, employed interventions, and used perceived learning to assess learning.

Fig. 10
figure 10

Map** of learning goal and applied type of learning assessment

Full size image
Table 5 Overview on the connection of learning goal, intervention and assessment of learning
Full size table

5 Discussion

To investigate the learning potential of hackathon-like events, we performed a systematic literature review focusing on three aspects: a) the promotion of learning in hackathon-like events, b) the orchestration of learning in hackathon-like events, and c) the standardization and systematization of learning in hackathon-like events. Here, we answer our three research questions based on our findings and elaborate on the theoretical and practical implications and contribution of this work.

5.1 How do hackathon-like events promote learning? (RQ 1)

Collins (2023) defines promote as "if people promote something, they help or encourage it to happen, increase, or spread". Therefore we interpret this research question as: how do the circumstances or activities within hackathon-like events actively help the participants to further improve their knowledge? In our investigation of the literature about learning in hackathon-like events, we found that not all papers mentioned an explicit learning goal for the described event(s). On the other hand, many organizers of such events aim for learning to happen, but they do not ground their work on appropriate theoretical foundations, for example, educational frameworks or learning theories, in their setup.

Furthermore, the assessment of learning, for most papers, is based on perceived learning. This suggests that there is the need for additional evidence to indicate the effectiveness of learning in hackathons.

From the students’ perspective, the short timeline of hackathons might lead to more superficial learning and would not necessarily be conducive to start learning something entirely new (Gama et al., 2018). Instead, students would rather stay in their comfort zone and stick with tools/methods they were already familiar with (Gama et al., 2021). This is something to be taken into consideration in future endeavors.

Taking the above into account, we argue that the promotion of learning in hackathon-like events is – as of now – a work in progress, and therefore, additional research on the topic is needed to establish the potential of hackathon-like events as learning opportunities.

5.2 How are learning activities orchestrated in the context of hackathon-like events? (RQ 2)

We found clear tendencies for individual event characteristics toward face-to-face events, informal learning, and using mentoring as an intervention in the reviewed literature. However, we also found cases where learning goals were set for the events, but no interventions were implemented or discussed. Overall we did not find a common schema that was followed in the setup of events. Such a schema – either in the form of a workflow or methodological framework – that would synthesize the intended learning outcomes, theoretical foundations, and appropriate setup for the target audience could guide the design of learning interventions and drive the assessment of learning outcomes. We envision that such a schema is necessary to take advantage of the full potential of hackathon-like events as learning opportunities.

Concepts like incidental or collateral learning proclaim that learning can practically happen alongside other activities and even while intending to learn something else (Dewey, 1953). Nonetheless, this does not mean that teachers or instructors with set learning goals for their students can just adopt a laissez-faire approach and rely on advantageous outcomes, as these are neither guaranteed nor predictable. For learning to happen, it is necessary to plan accordingly and design the activity specifically for learning (Kirschner & Hendrick, 2020). For hackathon-like events to work as intentional learning opportunities, we need to ask first what the concrete goal is, for example, improving a skill, learning new skills, gaining content knowledge about a new topic, or diving deeper into a familiar topic. Next, we need to examine the event characteristics that may promote or inhibit learning in the intended context and finally adjust the event set-up accordingly. For example, limited time and pressure to deliver a result can lead to participants relying more on established skills than learning something new, as seen in Gama et al. (2018). To that end, relevant instructional approaches, such as PBL can be used for guidance.

When moving hackathon-like events into formal education (for example, in a classroom), one has to reflect on the role and responsibilities of the teacher in tandem with the established roles within a hackathon, such as the mentor role. As most activities in formal education count for credits or grades, a classroom hackathon would likely be graded. If the teacher is responsible for grading, students might hesitate to ask questions if the person hel** them carry out their project is the same that will do the grading. Therefore, it might make sense to have someone else – other than the teacher – fill the mentor role. The options for this are manifold, for example, students from higher grades or – leaning more toward expert coaches – alumni or professionals. Beyond grades, it is also necessary to be mindful of organizational aspects. In formal education, the teachers and instructors are responsible for their students’ learning. What implications does this have regarding the organization of hackathons? For example, should teachers be organizing hackathons themselves, or should expert hackathon organizers do that? The latter may support the argument that hackathons appeal to informal learning instead. These aspects should be considered and studied further to explore the implications of the integration of hackathon-like events in formal education and to investigate how this could affect formal education dynamics and mechanisms.

5.3 How are knowledge and learning outcomes standardized and systematized in the context of hackathon-like events? (RQ 3)

For RQ 3 – looking into the standardization and systematization of knowledge and learning – we looked into the relation of the assessment of learning and learning goal, used inventions, and reported results. The most prominent finding was the predominant approach to assess learning by measuring perceived learning through self-reports. While individual studies reported mainly ad-hoc approaches in investigating learning, we did not find many holistic approaches in formalizing the learning process in hackathon-like events, from defining learning goals to planning and implementing appropriate interventions to assessing that the intended learning took place. As detailed by Chounta et al. (2017), the absence of standardization of learning and knowledge means that it is not possible to predict learning outcomes for these events. Determining a toolset to assess learning in these events would go a long way to identify a baseline of learned skills and help evaluate the effects of different interventions to facilitate intentional application.

5.4 Theoretical and practical implications

In this work, we pointed out a lack of consideration for educational components in how the set-up of hackathon-like events is reported. Often, learning in hackathons seemed to be taken as a foregone conclusion and as something that may happen or not, with no evident regard for the necessary, intentional set-up of a learning environment. We argue that it is imperative to bridge the gap between instructional theory and practice when setting up hackathons for learning to explicitly promote learning through those events.

What struck us most in our review was the number of reported studies that used self-reported perceived learning as an assessment tool to draw conclusions on whether learning took place or not. Related research points out that using perceived learning as an assessment for learning is controversial. Persky et al. (2020) found that "students are poor judges of their own learning" as the result of a study where the authors compared students’ self-assessment pre- and post-reading of information on their understanding of a topic, as well as on their perceived improvements, with the results of pre- and post-intervention quizzes. Contrary to this, Kuhn & Rundle-Thiele (2009) found a moderate association between perceived learning in a course and the course grade. However, the course design entailed four assignments that were all part of the final grade, and the set-up of the reported study having the self-assessment toward the end of the semester meant that the students had already received individual feedback on three assignments. This may suggest that the students didn’t rely on introspection alone for their self-assessment. Therefore, one could argue that this case is not comparable to the hackathon-like events we are looking at, as the study’s findings regarding the accuracy of the self-assessment may not be the same without receiving prior feedback. Sitzmann et al. (2010) also found a moderate relationship between self-assessment of knowledge and cognitive learning; they additionally found a strong relationship between self-assessment of learning and affective evaluative outcomes, which encompasses learner motivation and self-efficacy. This leads us to conclude that perceived learning – while valuable in its own right – might not be the best choice to evaluate learning in hackathon-like events if the goal is to use these events for learning.

Furthermore, since perceived learning can be unreliable as an assessment for learning, we see a need for more formal evaluation methods to assess the capabilities of hackathons to promote learning and, consequently, to be used with that aim. This should also include further investigation of individual interventions and their impact on learning.

Finally, we argue that further research is needed to gather insights and to solidify the role of hackathons with respect to learning. We envision that the proposal of a framework (or else, schema) to bring together design aspects of hackathon-like events and educational theory and practice is necessary to support contextualization and, consequently, establish the role of such events as learning opportunities.

6 Conclusion

In this paper, we presented an analysis of the state-of-the-art in the field of hackathons and hackathon-like events that aim to learning. For our work, we followed the Kitchenham guidelines (Kitchenham, 2004) cross-validated with the PRISMA statement (Page et al., 2021). Overall, we reviewed 39 papers published within 10 years, from 2012 to 2022.

Our findings suggest that learning in hackathon-like events is a point of interest for hackathon organizers and participants alike: learning is referenced as a motivation for both organizers and participants. Furthermore, hackathon-like events are frequently used in formal education environments. Research in the field studies learning as an outcome of these events but these investigations focus mostly on perceived learning. However, to what extent perceived learning can provide accurate assessments of learning outcomes in this context is questionable. Additionally, we found no evidence of systematic or standardized approaches when it comes to defining learning goals, the use of learning designs, or the evaluation of learning aspects in general. We also failed to identify in existing works implications regarding the role of teachers and instructors or the adaptation and adoption of hackathon-like events in formal education. Our suggestion for future work is to establish a systematized practice of assessing learning in hackathon events in order to lay a sound foundation for future research. This would also help in ultimately establishing research-based guidelines for hackathon organizers to optimize hackathons for learning.

6.1 Limitations

We acknowledge that this research is subject to limitations, which can serve as pointers for future work. The SLR has focused on research published within a specific time frame (2012 - 2022) and only in English. While we strive for completeness and inclusion, we acknowledge that any research beyond the specific time frame and publication language has not been included.

To conduct this SLR, we followed established guidelines (Kitchenham, 2004), and for every step, we cross-checked with the PRISMA statement (Page et al., 2021). Nonetheless, we acknowledge that we made specific decisions that may have biased the final selection of papers. We derived the list of names for hackathon-like events from related literature and through trial. We acknowledge that due to the amount of (creative) names that are used for hackathon-like events, it is possible that we have missed research on similar events during the search phase that did not reference our search terms.

6.2 Contribution

We envision that the contribution of this work is twofold; first, our work provided insights regarding the research landscape in the growing field of hackathons and hackathon-like events. Most importantly, our findings suggest that learning in hackathon-like events has been neither exhaustively nor systematically investigated. Second, we pointed out the need for future work, especially toward the direction of systematization and standardization of knowledge and learning in and from hackathon-like events, and the robust evaluation of learning outcomes. We argue that this work points to actionable insights for hackathon researchers to investigate further how hackathons can be used as intentional teaching methods and to pursue the collection of evidence that indeed hackathons can offer opportunities for learning.