Keywords

1 Introduction

The global gaming console market is expected to witness high growth over the forecast period [1]. In order to make gaming experience more enjoyable and win more customers, companies need to take usability into consideration when designing products. While many standard usability testing procedures and doctrine may not be applicable to video game design [2], we focus more on the usability of products themselves instead of video games.

Handheld game consoles contain the console, screen, speakers, and controls in one unit, which allow people to carry them and play them at any time or place. Playing handheld game consoles offers users different experience compared to PC or mobile games and can make playing games more active and flexible [9,10,11]. The first handheld electronic game can be dated back to 1976, when Mattel released Auto Race and the oldest true handheld game console with interchangeable cartridges is the Milto Bradley Microvision in 1979. Currently, Nintendo dominate the handheld console market.

Our study consists of two main experiments. In the first experiment, we made a comparative usability analysis of Nintendo Switch (Fig. 1) and Nintendo 3DS (Fig. 2), which are both Nintendo’s star-products while Switch, the later-released product, enjoys a better sale currently [3]. Switch is a portable hybrid console that merge the handheld and home console experiences while 3DS is a handheld game console. We only compared the handheld mold of Switch to 3DS and test task match, the ease of learning, and the ease of use respectively.

Fig. 1.
figure 1

Switch is a video game console developed by Nintendo and released on March 3, 2017.

Full size image
Fig. 2.
figure 2

The Nintendo 3DS is produced by Nintendo and is officially unveiled in June 2010. The one that we analyze in the study is Nintendo 3DS-LL, which was released in July, 2012 in Japan.

Full size image
Fig. 3.
figure 3

Powering on

Full size image
Fig. 4.
figure 4

Entering the game

Full size image
Fig. 5.
figure 5

Unlocking the product

Full size image
Fig. 6.
figure 6

Detaching the handles from the product

Full size image
Fig. 7
figure 7

Fitting the handles to the handle holder

Full size image
Fig. 8
figure 8

Entering the game

Full size image
Fig. 9
figure 9

Using the joystick to move

Full size image

In the second experiment, we focus on testing Switch and compare task match, the ease of learning, and the ease of use of two gaming modes of Switch: two-handle mode and single-handle mode. Switch has two detachable handles attached to the main product and can be used with both two handles or only one. The operations of two handles are different from the operations of one handle, which may cause confusion to new users.

The aim of the study is to compare the usability of two different handle modes of Switch and of Switch and 3DS, as well as raise proposals and suggestions of designing video game consoles.

2 Methodology

The variables of the study are adopted from Leventhal and Barnes’ usability model [4], which contains the ease of learning, the ease of relearning, the ease of use, task match, flexibility and user satisfaction. Since many operations of the consoles vary from game to game, we only tested basic universal operations which did not involve flexibility. We learned from pilot experiments that there was no apparent difference regarding relearning and users’ satisfaction, so we ruled out these three variables. The ease of learning was measured by perform variables of accuracy, and quantitatively rating scales as a supplement while task match and the ease of use were measured by quantitatively rating scales and qualitatively subjective opinions.

We also conducted the method of ‘Thinking Aloud Protocol’. According to Nielsen, this valuable usability test refers to the users’ narrative about the actions performed, the decisions made, their opinions and their feelings while interacting with the evaluated product (or prototype).

We recruited 12 users in total to participate in the two tests. They were all students aged from 19 to 21 in Hunan University and had no previous experience in playing Switch or 3DS-LL. They were asked to perform several tasks with Switch or 3DS-LL independently. If one failed to complete a task, he or she would be given standard instructions only once and be asked to try again. We recorded the performance of users’ every operation in the tasks and after they completed all tasks, we asked them to fill in a five-point scale rating table regarding the three variables to evaluate each operation and conducted semi-structured interviews to collect users’ opinions and feelings. During the experiments, a facilitator facilitated the tests, a recorder recorded and wrote down users’ words and opinions, another recorder shoot videos, and an observer observed users’ operations. The recordings and videos were recorded by two smartphones(Huawei Mate10 and Huawei Mate 10 Pro) respectively.

2.1 Test One

The first test was a controlled experiment [6, 7] to evaluate the usability of the handheld mode of Switch and 3DS-LL. We recruited 3 males and 2 females in the test. The pre-test hypothesis [6, 7] was that the overall usability of Switch when playing games was higher than 3DS-LL for college students. The independent variable was Switch or 3DS-LL while the dependent variables were task match, the ease of learning, and the ease of use. The sequence of tasks, the environment, instructions and machine conditions were rigidly controlled during different users’ testing.

The four tasks that users needed to perform in the experiment were listed below.

  1. 1.

    Powering on the evaluated product. Users needed to identify the power button and pressed it to power on the product.

  2. 2.

    Entering the game. Users needed to operate the joystick to choose the game we required.

  3. 3.

    Performing four main gaming operations: using the joystick to move, pressing the attack button to attack, pressing the shield button to shield and pressing the catching button to catch.

  4. 4.

    Operating to pause the game. Users needed to recognize the menu button and press it.

Nearly all buttons on the products are tested. During the test, users were encouraged to speak out their feelings and opinions towards the product and after which, they were asked to fill in the rating scales. Table 1 shows some example rating scales and questions for each operation are the same. After filling the scales, an interview about the usability of evaluated products was held to collect opinions and feelings. The goal of the experiment was to evaluate the usability of the same operations of Switch and 3DS-ll respectively and make a comparison.

Table 1. Sample questions from the rating table of the first experiment
Full size table

2.2 Test Two

The second test aimed to evaluate the usability of Switch with a focus on the use of its handles. It contained an experiment to test some basic operations of the handles and a controlled experiment to comparatively test the usability of the two-handle mode and single-handle mode of Switch. Users (4 males and 3 females) were asked to perform the same tasks with two handles and one handle respectively after standard instructions. The pre-test hypothesis of the controlled experiment was that the overall usability of Switch’s two-handle mode is higher than single-handle mode when playing games for college students. The independent variable was playing Switch with two handles or one while the dependent variables were task match, the ease of learning, and the ease of use. The sequence of tasks, environment, instructions and machine conditions were also rigidly controlled. Similarly, the ease of learning was measured by perform variables of accuracy and quantitatively rating scales, while task match and the ease of use was measured by quantitatively rating scales and qualitatively subjective opinions. Also, users’ words, operations and facial expressions are recorded.

The tasks that users were asked to complete in the second test were as below.

  1. 1.

    Unlocking the product. Users needed to identify the unlocking button and press it.

  2. 2.

    Detaching the handles from the product. Users needed to press a certain button and taking the handles away from the console.

  3. 3.

    Fitting the handles to the handle holder.

  4. 4.

    Entering the game with two handles. Users needed to operate the joystick to choose the game we required.

  5. 5.

    Performing the four gaming operations same to the first experiment with two handles.

  6. 6.

    Cutting the connection between handles and the product. Users needed to recognize the connection button and pressed it.

  7. 7.

    Connecting one handle to the product. Users needed to recognize the two connection buttons and pressed them simultaneously.

  8. 8.

    Fitting the single handle to another handle holder.

  9. 9.

    Entering the game with one handle.

  10. 10.

    Performing the four main gaming operations same to the first experiment with one handle.

The tasks 4), 5) are comparative to the tasks 8), and 9). After completing all the tasks, the users were given rating scales to rate the understandability, ease of learning and comfort of each tested operation as well as the overall understandability, ease of learning and comfort of the two-handle mode and single-handle mode. Table 2 shows some example rating scales and questions for each operation are the same. After filling the scales, an interview was held to collect more subjective opinions and feelings from users. Nearly all buttons on the products were tested. This experiment aimed to analyze the usability of basic operations related to handles of Switch and compare the usability of the two-handle and single-handle mode regarding the operations of entering the game and performing four main gaming operations.

Table 2. Sample questions from the rating table of the second experiment
Full size table

3 Results

The problems identified were categorized by the variables: task match, the ease of learning, and the ease of use. Regarding the quantitative results obtained in the first usability experiment, the average overall rating of Switch’s ease of learning is 4.20/5, while the ease of understanding and comfort is 4.40/5 and 4.00/5 respectively. Generally, subjective ratings present a similar trend with users’ objective task performance and serve as a strong supplement. The average overall rating of 3DS-LL’s ease of learning is equal to Switch (4.20/5), while its ratings of the ease of understanding and comfort are around 0.5 points lower. The ratings of powering on, entering the game, pressing the ‘a’ button to attack and using joystick to move are relatively high (all average ratings of the three variables are over 4.00/5) and are similar between the two products, while the pausing operation of both products receive relatively low ratings in the three variables (most are around 3.00/5).

About the first variable, task match, the operations of 3DS-LL were better understood by the users. An apparent problem of Switch is that none of the tested users (100%, n = 5) could understand the metaphor behind the ‘+’ button, the one to call up the menu. One participant mentioned that the ‘+’ button is small and not salient enough, making it looks like an ornament instead of a button. He also suggested that a round shape would make it more like a ‘button’. Additionally, it is reported that the corresponding ‘−’ button also brings great confusion since it has no clear function, unlike ‘+’. Additionally, the average ratings of shielding and catching of Switch are lower than 4.00/5). It is largely because that Switch has four similar oblong buttons on the top for catching and shielding (while 3DS-LL only has 2), which made users’ cognition load heavier and caused confusion. Other operations of Switch are easy to understand generally. As for task match of 3DS-LL, the main problem is that the average rating of pausing is only 2.80/5 and one participant mentioned that the menu button for pausing is not obvious enough.

Regarding the second variable, the ease of learning, feedbacks are generally good. All ratings of Switch and 3DS-LL are over 4.00/5 except the pausing operation of Switch and all users (100%, n = 5) finally completed all tasks successfully after receiving standard instructions. Users’ opinions towards the ease of learning of both products are optimistic, one user mentioned that ‘even some operations may not be easy to understand, they are basically easy to learn’. Nevertheless, three users (60%, n = 3) failed to complete the task of ‘pausing the game’ with Switch and two users (40%, n = 2) failed to complete the same task with 3DS-LL at the first time.

On the third variable, the ease of use, several problems emerged. For Switch, two users (40%, n = 2) mentioned that the powering button of Switch is a bit small for their fingers to press. Two male users (40%, n = 2) mentioned that the oblong buttons on the top for catching and shielding are too close to each other while the four buttons for gaming operations on the front of Switch are also in close proximity, which may result in their mistaken triggering. As for 3DS-LL, three users (60%, n = 3) reported that the feedback of the menu button is ambiguous, which made them question whether they press the button successfully and one user (20%, n = 1) mentioned that the powering button of 3DS-LL is small and the surface is a bit concave, which makes it difficult to press. Interestingly, users’ opinions vary on the joysticks. Two users (40%, n = 2) showed apparent preference to the joystick of Switch while another two (40%, n = 2) loved the joystick of 3DS-LL. The reasons for choosing Switch’s joystick are that it is more comfortable to touch, its control and orientation is more acute and effortless (especially when performing two successive movements towards the same direction), and it gives users a strong feeling of operating since it likes a plane rocker. However, the reasons for preferring 3DS-LL are that the joystick has larger dam** and can control the movement more accurately while the joystick of Switch is too acute and slippery to operate.

Regarding the first experiment of basic operations of Switch’s handles in the second test, the rating of ‘detaching handles from Switch’ on test match is the lowest (2.57/5) and three users (43%, n = 3) rated lower than 3. Two participants (29%, n = 2) first thought that handles should be pulled horizontally to both sides while they actually need to be pulled vertically. The rating of ‘fitting the handle to the single-handle holder’ is the second lowest (3.00/5) and all participants (100%, n = 7) installed the single handle upside down without noticing the hint provided by the product. Another two operations rated lower than 4 are ‘cutting the connection between handles and the product’, and ‘fitting the single handle to the single-handle holder’. When performing ‘cutting the connection between handles and the product’, one user said that the cutting-off button is too hidden and small, making it unlike a functional button, while the blue ornamental salient cube next to it is more like a functional button. It is also worth noting that when completing ‘fitting the handles to the handle holder’ two participants (29%, n = 2) tried to press the handles on the handle holder instead of inserting the handle into the handle holder. Besides, when doing ‘entering the game’, one user reported confusion since the joystick is used to choose instead of the up, down, left and right buttons.

As for the ease of learning, results are also generally optimistic, except ‘detaching handles from Switch’ and ‘fitting the handle to the single-handle holder’ (3.57/5 and 3.86/5 respectively). Users’ task performances are consistent to the ratings since two users failed three time when completing ‘detaching handles from Switch’ and four users failed one or two times when completing ‘fitting the handle to the single-handle holder’.

On the third variable, the ease of use, all operations receive relatively low ratings (all lower than 4/5 except ‘connecting one handle to the product’). The lowest average rating is ‘unlocking to product’ (2.86/5) since it requires users to press any button three times and all participants (100%, n = 7) believed that there is no need to do so. The second lowest is ‘detaching handles from Switch’ and three male users (43%, n = 3) all reported that the button for detaching is too small to press and one female user mentioned that the operation requests great efforts. Besides, two users (29%, n = 2) reported that the button for ‘cutting the connection between handles and the product’ is too small and hidden, making it difficult to press. Additionally, three users (43%, n = 3) held that the positions of buttons and the joystick on the handles of Switch are biased to one side, making it uncomfortable to use with a single handle.

As for the compared experiment of the second test, generally, the ratings of the single-handle mode are higher than the two-handle mode. On the first variable, task match, the ratings of ‘catching’ and ‘shielding’ of both modes are relatively low, corresponding to the first experiment. About the two-handle mode, the average rating of ‘catching’ with two handles is 3.57/5 and four participants (57%, n = 4) rated 2 for ‘shielding’. Regarding the single-handle mode, the average rating of ‘attacking’, ‘catching’, and ‘shielding’ all equal 4.14/5. Two novice users (29%, n = 2) reported that the different functions of the same buttons between the two modes caused their confusions. On the second and third variables, all ratings are above 4/5 except the ‘catching’, and ‘shielding’ operation of the two-handle mode. Similar to the first experiment, it is largely because the four similar oblong buttons on the top of Switch’s handles (two on each handle), which results in confusion and inadvertently triggering. However, when using the single-handle mode, there are only two oblong buttons on the top, which reduces users’ cognitive load and makes these two operations easier to understand and operate with one handle.

4 Limitations and Discussions

We analyzed the usability of Switch and 3DS-LL and received many opinions and suggestions from the users. However, there are some limitations of our study:

  1. 1.

    The tested users are all aged from 19–21 while Switch and 3DS-LL are designed for a much larger range of age.

  2. 2.

    When doing the compared experiment in the second test, the tasks of the two-handle mode and the single-handle mode were the same. However, the basic operating rules of the two modes are similar and users completed operations with two handles before one handle, which may make the performances of the one handle mode better. This may partly explain why generally, users rated higher scores of the single-handle mode than the two-handle mode, while they prefer using two handles instead.

Discussions that require further research are listed below:

  1. 1.

    One participant reported that the round salient point above the screen of 3DS-LL seemed like a powering button to her. However, this may not be changed because the two points served as shock absorbers.

  2. 2.

    The positions of buttons and the joystick on the handles of Switch are biased to one side, making operating with single handle uncomfortable. However, this may not be changed because it is to compromise the usability of the two-handle mode, which is a more common mode.

  3. 3.

    Two users mentioned that the functions of the same button are different when shifting between the two-handle mode and single-handle mode of Switch, which caused their confusions. However, this is also hard to fix since the handle is used horizontally when using the single-handle mode while the handles are used vertically with the two-handle mode, which change the buttons’ relative position.

  4. 4.

    Two male participants thought the four oblong buttons on the top of Switch and the four functioning buttons on the front of Switch are in close proximity and are easy to trigger one or another mistakenly. However, they are all male adults above 175 cm in height while users of Switch also include kids.

  5. 5.

    All participants preferred to use two handles of Switch because they thought the single one handle was too small for their hands. However, they are all adults above 155 cm in height while users of Switch also include kids.

5 Recommendations

There are also more general problems and practical suggestions we would like to give to the design of Switch and 3DS-LL regarding usability. The recommendations for Switch are as below:

  1. 1.

    Make the powering-off button of the handle bigger and more salient.

  2. 2.

    Make the powering-on button of the product bigger and more salient.

  3. 3.

    Make the direction symbols for handle installation clearer.

  4. 4.

    The four oblong buttons on the top can be discriminated by different textile.

  5. 5.

    Add symbols to hint that the handles should be pulled vertically in order to be detached from the product instead of being pulled vertically.

  6. 6.

    Make the ‘+’ and ‘–’ buttons more salient or change their shapes to round, enabling them to look more like functional buttons.

The recommendations for 3DS-LL are as below:

  1. 1.

    Make the tactile feedbacks of the three buttons below the screen clearer.

  2. 2.

    Two touch screens of 3DS-LL may make users feel confused and dizzy. Reducing to one screen should be considered.

6 Conclusions

Despite the success of Switch in the handheld game consoles’ market, several still needs to be improved in order to ensure a better gaming experience for users, especially for adult users. Since Switch intended not for the children but also for adults, results from our study are valuable references. Most problems of Switch and 3DS-LL obtained from the study regarding the ease of use and task match while the level of ease of learning is relatively high. It implies that as game consoles, Switch and 3DS-LL’s basic operations are generally easy to learn, which is good to novice users. However, task match and the ease of use of both products should be improved.