Empirical approaches to quantifying effort, fatigue and concentration in the conceptual design process

Abstract

Recently, physiological signals such as eye-tracking and gesture analysis, galvanic skin response (GSR), electrocardiograms (ECG), and electroencephalograms (EEG) have been used by design researchers to extract significant information to describe the conceptual design process. We study a set of video-based design protocols recorded on subjects performing design tasks on a sketchpad while having their EEG monitored. We propose empirical approaches to quantify effort, fatigue, and concentration during the conceptual design process. To perform this analysis, we extract EEG features that convey information on effort, fatigue, and concentration. We argue that all three are relevant in the conceptual design process. Such an analysis has the merit of being fully automated, readily integrable in engineering systems and not being subjected to the subjectivity of the domain expert performing the analysis like in subjective rating frameworks. Our analysis leads to four hypotheses: (1) effort and fatigue are subjected to ice-breaking and end of task phenomena; (2) fatigue and effort follow a capacity model; (3) fatigue is multidimensional; and (4) concentration follows a modal shift model.

Appendices

Appendix 1: EEG analysis

1.1 Spatio-temporal features: microstates

D. Lehmann, C.M. Michel, Koenig, R.D. Pascual-Marqui et al. in a series of papers Lehmann (1971, 1990), Lehmann et al. (1987), Pascual-Marqui et al. (1995) and Koenig et al. (1999, 2002) developed the notion of functional microstate analysis of the brain based on machine learning algorithms over the spatio-temporal domain of EEG signals gathered from known locations on the scalp. Lehmann (1990) describes the latter as being the atoms of thought. Functional microstates of the brain are prototypical patterns of the scalp field maps and, in terms of pattern recognition, can be seen as the cluster centroids of an EEG signal (cf. Fig. 12). Large-scale statistical studies have shown that most humans have the same four microstates when EEGs were measured on subjects at rest with eyes closed. Microstates have been shown to vary depending on age groups (Koenig et al. 2002). Microstates constitute the main technique of spatio-temporal analyses of EEG signals (Michel et al. 2009; Brunet et al. 2011).

Fig. 12

Four prototypical microstates computed using the P2ML algorithm from an EEG epoch, where the subject was asked to close his eyes

In the following, we describe the two P2ML algorithms (Pascual-Marqui et al. 1995). For a set of k given microstates $M_k$ and potential values $V_t$ at time t, the P2ML objective function is

$$\begin{aligned} {\mathrm {arg}}\ \underset{k}{{\mathrm {max}}} \{(V_t \cdot M_k)^2\}. \end{aligned}$$

(1)

Using vector geometry (Freudiger 2003), the orthogonal squared distance between $V_t$ and $M_k$ is

$$\begin{aligned} d^2(V_t, M_k) =(V_t \cdot V_t)-(V_t \cdot M_k)^2. \end{aligned}$$

(2)

The objective function is effectively minimizing the orthogonal squared distance between $V_t$ and $M_1,\ldots ,M_k,\ldots$ by maximizing the $(V_t \cdot M_k)^2$ term of the equation.

The regularized objective function of the P2ML method is

$$\begin{aligned} {\mathrm {arg}}\ \underset{k}{{\mathrm {min}}}\left\{ \frac{( V_t \cdot V_t - ( V_t \cdot M_k)^2)}{2e(N-1)}-\lambda E \right\} \end{aligned}$$

(3)

where $\lambda$ is a smoothness penalty coefficient and, N is the number of samples, and E is a smoothness penalty function given by

$$\begin{aligned} E=\sum _{i=t-w}^{t+w} \delta (S_i,n),\quad n=1,\ldots ,k \end{aligned}$$

(4)

for a segment number $S_i$ (the numerical index k of the microstate $M_k$) and a delta function given by

$$\begin{aligned} \delta (x,y)= \left\{ \begin{array}{ll} 1,&\quad x=y \\ 0, &\quad \text{otherwise}. \end{array} \right. \end{aligned}$$

(5)

The regularization parameter is $\lambda E$ and can be seen as a smoothness penalty factor. It increases when segments are long and decreases when segments are short based on a window size parameter w. Therefore, the regularized objective function has low values for smooth segments and high values for non-smooth segments, effectively penalizing transient microstates.

Following the definition of the regularized and non-regularized microstate algorithms, the transient microstate percentage (TM%) can be defined as follows (Nguyen et al. 2015b):

$$\text{TM}\%=\frac{({\mathrm{{Segments}}}-{\mathrm{{SmoothSegments}}})}{N}.$$

(6)

The transient microstate percentage is a significant feature. In design research, it is used in Nguyen et al. (2015b) to segment design protocol data using EEG. In Nguyen et al. (2015a), it is used as a correlate to mental effort.

For example, let an EEG signal have 10 samples. The non-regularized segmentation of an EEG signal based on four microstates could be: (1, 1, 1, 1, 3, 3, 3, 2, 4, 4). The sample segmented to microstate 2 has a transient duration of only one sample. Assume that the regularized segmentation is then: (1, 1, 1, 1, 3, 3, 3, 4, 4, 4). The segment at index 8 has been smoothed out and the associated transient microstate percentage is $10\%$.

1.2 Frequency-domain features

Frequency-domain features are typically computed on the Fourier transforms or the wavelet (multiresolution) transforms of an EEG signal. In EEG analysis, a popular and common method is based on power spectral densities (PSD). The usual frequency ranges in neurosciences are the alpha band, the beta band, the delta band, and the theta band. Frequency ranges (that may change depending on authors) are given in Table 3. Furthermore, combinations of frequency ranges are used by researchers to narrow down different characteristics such as fatigue and concentration (Jap et al. 2009).

$$PSD_{range} = \sum_{\tau \in range} \big\vert(DFT_\tau(\textbf{x})) \big\vert$$

(7)

In contrast to microstates (which are computed on multichannel measurements of an EEG), PSD is usually computed on a given electrode such as FP1 (left frontal area). PSD is a one-dimensional method and has been widely used to detect eye-blinking artefacts, sensorimotor artefacts, sleep cycles and states of mind such as fatigue, concentration, and relaxation (Blankertz et al. 2010).

Table 3 EEG frequency-domain features (Baumeister et al. 2008; Jap et al. 2009)

Appendix 2: A mathematica implementation of P2ML and transient microstates calculation

The code in Listing 1 is a Mathematica implementation of the P2ML algorithm. It requires as input a data matrix of dimensions $N\times M$, where N is the number of samples and M is the number of electrodes, a number of clusters (e.g., 4) and a convergence parameter $\epsilon$ (e.g., 0.0001). In addition, it accepts the following options: MaxIterations (the maximum number of iterations allowed), Verbose (print execution details), and Microstates (provide as output parameters the microstates in addition to the segmentation).

The original P2ML algorithm is sensible to data lengths as it expects all clusters defined in the input parameter to be represented in the data. This occurs with higher probability on longer epochs. Experimentally, an epoch length of 2500 samples at a data rate of 500 samples per second is usually enough. Future work could handle this error case.

The code in Listing 2 is a Mathematica implementation of the P2ML regularized smoothing algorithm. It requires a data matrix of the same dimensions as above, a number of clusters (e.g., 4), a convergence parameter $\epsilon$ (e.g., 0.0001), a smoothness penalty parameter $\lambda$ (e.g., 5), and a window size parameter w (e.g., 3).

A sample usage to compute the transient microstate percentage is in Listing 3. It includes the package we developed for the P2ML and regularized P2ML algorithms.