Keywords

1 Introduction

Museums and archives possess significant amounts of cultural heritage data that often remain unused or merely receive little attention. One reason for this is the lack of suitable interfaces for exploration. Real value is created by making this data accessible to the public through suitable interfaces for exploration and discovery. In this paper, we describe an applied research based on the Ringier Bildarchiv (RBA), a large database of historical photographs. Our main objective is to engage the general audience (i.e. non-expert users) in the exploration of this data and thus facilitate discovery and insight while also providing a great user experience. In the context of museums, immersive visualisation combined with embodied interaction  [1] is offering promising prospects for that purpose when implemented with vision based technology, where no contact between user and hardware devices is required. And in comparison to other immersive technologies such as virtual reality, MR output is highly visible and attractive for surrounding museum visitors. An additional benefit of this method notably for large databases lies in its ability to rapidly offer an overall impression of the contents.

The RBA containing more than 7 million analog press photos is Switzerland’s largest picture archive. The images document people and events in the country’s recent history (1930–2000) and cover a wide range of public and private life matters – from politics to sport and culture to everyday life. In the course of digitalisation, the analog archive originally collected and curated by a prominent publishing house was handed over to the state archive. Since 2015, sections of the collection are accessible to the public at the “Schauarchiv” (engl. archive exhibition) of Stadtmuseum Aarau where – under professional supervision and instruction on proper dealings with these historic contents – users can browse through the original archive boxes. In addition, selected portions of the RBA have been digitised and systematically stored in a database and public access is offered via a web interface  [2]. Transforming this ordinary online collection into an interactive exploration space for museum visitors offers an interesting use case for this research.

2 Theory and Related Work

Idreos et al. define data exploration as “efficiently extracting knowledge from data even if we do not know exactly what we are looking for” and explain that some key facets to achieve this objective are advanced data visualization and alternative exploration interfaces that help users navigate the underlying data space  [3]. In this research we focus on the latter. Interactive immersive technologies provide such an alternative interface that stimulates user engagement and cognition. Already in the 60s, Sutherland illustrated in his visionary essay “The ultimate display” how technology can augment human cognition and take advantage of a broader range of senses and capabilities than tradition interfaces  [4].

Interactive immersive technologies offer multiple benefits for engaging users in the data exploration task: Direct manipulation has been lauded for the reduction of information processing distance between the user’s intentions and the facilities provided by the system. This reduces physical as well as mental work and lets users accomplish their goals with less effort  [5]. In regards to data exploration, LaViola et al. claim mounting evidence that direct interaction offered by immersive environments provides benefits for perception and interpretation of data. They argue that such direct manipulation interactions are more fluid and more efficient in comparison to non-immersive data displays with traditional WIMP interaction  [6]. Additionally, affordances offered by the established relation between input and output vocabularies not only promise a faster learning curve but also contributes to the sense of immersion. Cummings further argues that greater immersion produces a stronger sense of presence which leads then to a higher engagement in the system or experience  [7].

The theory of embodied interaction unites immersion into the virtual environment with direct, natural user interaction and adds a phenomenological motivation for exploring psychological and social aspects of human-computer interaction. Dourish defines embodied interaction as “interaction with computer systems that occupy our world, a world of physical and social reality and that exploit this fact in how they interact with us”  [1]. Embodiment super-positions abstract data spaces with concrete, tangible interaction spaces and by doing so, harnesses the potential of mental processes that are embedded into the body  [8]. And since the human experience is shaped by the physical world, it’s not surprising that we are most engaged and effective when acting physically. Embodiement is further closely coupled to topics such as proprioception and spatial memory, that describe how we perceive and understand our bodies and our environment. Concerning the latter, embodiment and the phenomenological movement emerged amongst other influences from Uexküll’s concept of “Umwelt”. This theory originating from the field of biosemiotics maps out the spheres of perception of an individual  [9] and is primarily concerned with the user’s natural environment, not the technology involved. Weiser who coined the term “calm computing” anyhow campaigns that technology should be invisible and not intruding the user’s consciousness  [10]. This is particularly true for the given use case, where the user’s focus should be guided on the content and not on the technology itself. And Elmqvist et al. offer some guidelines for “fluid interactions” where they emphasise, inter alia, the importance of ensuring that continued exploration is always possible and users never reach dead ends  [11].

While many museums offer online collections to explore their databases, data exploration by the means of interactive immersive environments are still rare in practice in the cultural heritage domain. Nonetheless, an interesting example is mARChive. This interactive 360-degree data landscape offers an intuitive platform to engage with a collection of 100’000 objects of Melbourne’s Museum Victoria  [12]. Also noteworthy, from a related domain, is Müller et al.’s zoomable UI, a physically-based data exploration approach – inspired by the search process of a rummage table – to interact with data by deforming the surface of an elastic display  [13]. An example showing that embodied interaction leads to higher user engagement is “Be the data”, which pursues the objective of educating non-STEM students about data analytics using an embodied approach. Findings indicate that this method provides the necessary engagement to enable students to quickly learn about high-dimensional data and analysis processes despite minimal prior knowledge  [14]. And for gestural interfaces, Minority Report is a prime example. Yet before being scientific advisor on the film, Underkoffler developed Oblong g-speak, a spatial operating environment offering gesture control and multi-user collaboration for big data analytics  [15]. One of it’s applications particularly relevant for this research is “Exo”, a visualisation tool for exploring nearly 2300 exoplanets identified by NASA’s Kepler mission  [16].

3 Methods, Process and Materials

With the goal of animating non-expert users to explore the contents of the RBA archive in mind, we investigated various visualisation and interaction possibilities, including embodied interaction, and technological capabilities for recognizing user input in MR. We examined the spheres of perception and their reciprocal effect for the given use case (see Fig. 1). With respect to the museum use case interaction concepts, that do not require direct contact between user and hardware, and render output in a way that is relevant not only for the main user but also for the broader audience, were favoured.

Fig. 1.
figure 1

Umwelten

Full size image

We specified interaction patterns required to reach the goal. Following Shneiderman’s principle “overview first, zoom and filter, then details on demand”  [17] the application should allow interaction with the archive (or a subset) as a whole as well as allowing to zoom in on items of interest and provide additional details on selected objects. In addition, we defined levels of involvement ranging from 0 to 3 (no active participation to high user involvement) for a potential solution.

Applying design thinking we developed a set of concept ideas inspired by research on state of the art and technology. Concept evaluation considered factors such as dimensions of meaning, determinants regarding usability and user experience, feasibility and relevance for the use case. Selected concepts employing embodied interaction aspects were further developed by the means of rapid prototy** and resulted in the installation prototype described in Sect. 4.

To examine the effect of embodiment on the exploration task and single out this one factor – namely the form of interaction – for examination from all other conditions that may have an effect  [20], we designed a study that compares embodied interaction with WIMP interaction. To test the hypothesis, that embodied interaction stimulates users more to explore, make discoveries and gain insight from information spaces than WIMP interaction, we’ve built two prototypes that differ solely in the way how users interact. In one scenario, we use traditional WIMP interaction (i.e. mouse input). In the other scenario, we’ve applied the concept of embodied interaction by which the user’s body movement and position in the room as well as in-air gestures are used for application control. The installation contained two sets of 120 photos each, that were randomly selected from the RBA photo database and the setup is shown in Fig. 2.

Table 1. Research questions for user study
Full size table

We’ve conducted a user study (n = 16) where the form of interaction was the only independent variable in the study design. For dependent variables we defined metrics derived from user behaviour, performance and self declaration targeted to the 3 research questions (see Table 1). The test procedure comprised a pre-questionnaire (demographics, technical affinity, interest in application domain). Then for each scenario, first the features offered by the given prototype were demonstrated and there was time for learning the novel forms of interaction before doing the actual exploration task, followed by an evaluation of performance by means of questionnaire with open questions and ratings on 5-point Likert scales regarding immersion, fatigue as well as the two standardized evaluation scales UES-SF  [18] and UEQ  [19]. In addition, user behaviour was logged during the use of the application. The procedure ended with comparative and open questions and took about 60 min. in total. The participants (ages 21–71, ø 41, 10 male, 6 female) were of diverse occupational backgrounds. 80% indicated some interest in museums, 5 stated to have experienced immersive technologies previously and 7 tried in-air gestures once before (e.g. Nitendo Wii, Microsoft HoloLens). Only 2 declared occasional use of this form of interaction.

Fig. 2.
figure 2

Setup of the installation

Full size image

4 Results

We’ve developed an immersive interactive MR exploration space offering over-view, detail view and selection history as key features for exploring the contents of the image collection. In overview mode (see Fig. 3, left) objects are floating around freely. This fluid visualisation rendered on a large display should engage museum visitors to interact with the application and animate explorative behaviour. By physically moving around the interaction space, the user virtually moves through the depths of the super-positioned collection data space. Items that are further away from the user are rendered with lower brightness to increase depth perception. The application allows to zoom in on items of interest in an embodied manner by simply approaching them. By selecting an object – executed through an in-air click gesture – the application switches into detail view mode (see Fig. 3, right), where additional information for the selected item such as description, location, year and photographer is provided. In addition, a selection history is offered in this mode to return to previously selected objects for the purpose of comparison. Furthermore, the MR setup ensures that the application offers an attractive experience not only for the interactor, but also for others in vicinity who act as mere observers.

The software is implemented with Unity and the user’s gestures and movement are tracked through a depth camera (see Fig. 2). Although this computer vision based technology is less precise in recognising gesture input than body-attached sensors, it’s more suitable for use in public settings such as museums.

Fig. 3.
figure 3

Screenshots of overview mode (left) and detail view with selection history (right)

Full size image

Results of the user study confirm benefits of using embodied interaction with respect to all research questions. For exploration (Q1), logging data on virtual distance covered reveals that the collection space was explored to a significantly higher degree and the majority of users gauged this form of interaction to be more animating for the exploration task. Also UEQ’s stimulation score is considerably higher and attains the benchmark ‘excellent’. Concerning discovery (Q2) evaluation of user performance showed slightly more discoveries of novelties and correlations, plus UEQ’s novelty score is significantly higher. Besides, there was a stronger agreement on obtaining a fresh perspective on the contents. In regards to user engagement and user experience (Q3) UES-SF’s reward factor and aesthetic appeal scores were considerably higher and a higher sense of immersion was reported. There was no considerable difference in UES-SF’s overall user engagement score (3.79 vs. 3.71, on scale from 1 to 5), yet UES-SF’s perceived usability is significantly weaker. Overall, while scores for hedonic qualities were significantly higher (UEQ-HQ 1.67 vs. 1.18, on scale from -3 to +3), pragmatic qualities were considerably lower (UEQ-PQ 0.82 vs. 1.46). The primary reason for this were inaccuracies in gesture recognition. Although UEQ values above 0.8 are considered positive, this points to explicit potential for improvement.

Fig. 4.
figure 4

Participants interacting with the installation during the user study

Full size image

For this reason the depth camera was subsequently replaced with a higher quality device and a qualitative follow-up usability study (n = 5) using the new hardware was conducted  [21]. Results showed some improvement in UES-SF’s perceived usability score (3.60). There was no major change in the UEQ-PQ score, yet an increase was noticed in the UEQ’s overall attractiveness score (1.60), which demonstrates the interconnection of usability and overall user experience.

5 Discussion and Outlook

Both scenarios received rather high scores compared to UEQ benchmark data. It was particularly surprising, that even the WIMP interaction scenario received considerable positive ratings in particular by the older segment of participants who had in general more difficulties with gesture input. Several participants of this cohort mentioned that controlling gesture input absorbed their full attention. A reason for this could be that some research unveils spatial interaction problems as an early indicator for age-related dementia  [22]. While gesture input has many benefits such as the natural characteristics of direct map** of user movements, a drawback for all demographics is the need for clear delimiters to indicate initialisation and determination of gestures. Otherwise normal human motions may be interpreted as gestures while not intended as such. This concerns only in-air gestures, whereas the proposed embodied depth movement provides a continuous input and is therefore not affected by this issue.

In addition to the form of interaction also the personal interest in the application domain in general, and specific topics covered by the archive in particular, affects to what degree test users engage with the presented data. Differences in the overall interest in the application domain were taken into account and compensated for in the test design, however the personal emotional reference made with particular photographs such as emotions evoked by memories about a past incident are highly subjective and unpredictable.

Further research questions include investigation of the connection between spatial awareness, immersion and exploration, the impact of emotional references to the photograph’s content on explorative behaviour and how a shared experience and gamification approaches could increase the level of involvement and thus engage users even more in the exploration of historic photographic artifacts.