Time for Tele-TTO? Lessons Learned From Digital Interviewer-Assisted Time Trade-Off Data Collection

Abstract

The preferred mode of administration of time trade-off (TTO) in large-scale valuation studies is face-to-face (personal) interviews facilitated by a trained interviewer. Geographical, financial or situational constraints could complicate personal TTO interviews. When facing such constraints, the use of digital interviews, in which trained interviewers facilitate through videotelephony software (i.e. tele-TTO) may be considered. This paper aims to guide researchers in how to approach tele-TTO interviews and discusses their advantages and disadvantages. The main advantages of tele-TTO compared to personal TTO are decreased need for travel and increased flexibility of interview scheduling, which could reduce costs and may foster representative sampling. Possible disadvantages of tele-TTO are partial loss of visual cues, complications with building rapport and possible selection effects that result from differences in interview preparation. Furthermore, the paper reports on lessons learned from a project in which both personal TTO and tele-TTO interviews were conducted. The results of this project suggest that although they require a different recruitment and interview process, tele-TTO interviews are feasible and provide flexibility to the interviewer. Furthermore, tele-TTO interviews yield largely similar results. Future research should explore the role of possible selection effects and respondents’ perspective on tele-TTO interviews.

FormalPara Key Points

When geographical, financial or situational constraints complicate the use of face-to-face time trade-off (TTO) data collection, the use of digital interviews facilitated by trained interviews and videotelephony software is a feasible alternative.

Advantages of digital TTO interviews compared to face to face are the decreased need for travel and increased flexibility of interview scheduling, which could reduce costs and may foster representative sampling. Possible disadvantages of tele-TTO are partial loss of visual cues, complications with building rapport and possible selection effects that result from differences in interview preparation.

A comparison of personal and digital TTO interview data suggests that the results are similar for both modes of administration.

1 Introduction

The time trade-off (TTO) method, which requires trading off length and quality of life, generally produces a large amount of inconsistent or implausible responses [1]. To improve data quality, the methods used to operationalise TTO in large-scale valuation studies have developed substantially in the past decades [2]. Whereas early valuation studies relied on props to illustrate TTO [3], this valuation method is now often operationalised digitally. For example, the latest valuation protocol for EQ-5D-5L prescribes the use of computer-assisted face-to-face (personal) interviews (CAPI) for TTO using highly standardised software [4], although alternative software packages exist [45, 46], where such inequalities may be even larger than in OECD countries.

In this project, no data were collected on respondents’ or the interviewer’s experiences during personal- and tele-TTO interviews. As such, although it is seen as a key concern in qualitative work [19, 34], it is not possible to determine if the respondents and interviewer experienced differences in rapport. Weller [34] investigated this question in qualitative longitudinal research, in which videotelephony was introduced in a project with young-adult respondents who had relied on home visits for data collection in earlier phases. When comparing interview experiences, Weller [34] notes that the rapport experienced during these videotelephony interviews could be similar or at times even better (see also [17]). For example, some respondents experienced the interview as more personal and felt less pressure. However, this appeared to be dependent on the quality of the audio-visual connection. For example, whereas personal interviews often commenced by exchanging pleasantries (which are important for establish rapport), digital interviews often commenced with ensuring the connection was of sufficient quality [34]. Furthermore, when the audio quality impeded the interview, both interviewer and respondent had to invest considerable energy understanding each other and avoid interruptions, rather than thinking through their answers [34]. The latter seems of particular importance for tele-TTO, as in TTO where the use of mental shortcuts, rather than thinking through the answer, is often seen as a problem [47, 48]. Hence, further research should explore whether tele-TTO affects interviewer and respondents’ experiences, as it appears that these are heterogeneous [34]. It could also be explored if a hybrid data collection is feasible, in which the decision between tele-TTO and personal TTO is left up to the respondents. Such strategies were used successfully in qualitative research [17].

7 Conclusions and Additional Suggestions for Future Research

Admittedly, the results of this ongoing project should be carefully interpreted as it was not designed with the goal of comparing tele-TTO and personal TTO interviews. Hence, a study that has this explicit goal is a first paramount step for future research. Furthermore, future research may explore the role of alternative devices and software. Most tele-TTO interviews for this project were completed with a Zoom connection on a laptop. Perhaps the feasibility of tele-TTO interviews depends on the software used, as the options available to interviewers and the steps required for respondents to connect may differ. Performing tele-TTO interviews on mobile phones may require changes to the operationalisation of TTO (e.g. increased font size). Such changes could be worthwhile, as access to mobile phones and wireless connection is growing worldwide [49].

In conclusion, this paper suggests that tele-TTO may be a feasible, flexible, and low-cost alternative to personal TTO interviews, with some evidence suggesting that both modes of administration yield mostly similar outcomes. However, tele-TTO may suffer from selection effects and the use of videotelephony software may complicate the interview process. As situational constraints may prohibit the use of personal TTO interviews (e.g. COVID-19), additional research studying when, why, how, and for whom tele-TTO interviews should be used appears valuable.

Appendices

Appendix A Health states used for personal and tele-time trade-off interviews

All health states were derived from EQ-5D-5L, and as such utilise five domains: mobility, self-care, usual activities, pain/discomfort and anxiety/depression. The 5L version of the EQ-5D distinguishes five levels of severity on each domain, from ‘no problems’ to ‘extreme problems/unable to’. Health states are typically denoted by five digit codes such as 22113, with each number representing severity of the relevant domain. The Dutch translation of EQ-5D-5L was used, but below the health states are reprinted in English in Table 1.

Table 1 Health states used for tele-time trade-off (TTO) and personal TTO interviews

Appendix B Comparison of the outcomes and data quality of personal- and tele-TTO interviews

In this Appendix, the effect of mode of administration on TTO will be compared, by studying differences between tele- and personal TTO interviews in terms of decision outcomes and decision quality. Before presenting this analysis, some additional details are provided about the TTO method used.

2.1 TTO Method and Data

In this project, as is prescribed in the protocol developed for valuation of EQ-5D-5L [4], composite TTO was applied. This method, developed by [50], uses ‘conventional’ TTO to value health states considered better than dead and introduces a lead-time TTO to value health states considered worse than dead. This process was repeated for each health state described in Appendix A.

Conventional TTO tasks were operationalised with a 10-year duration. As such, it described to respondents a life in impaired health (e.g. a wheelchair) with a 10-year duration. Respondents were asked how many years in perfect health they found equivalent to this life (i.e. how many years the respondent would be willing to give up to live in perfect health for a shorter duration). If respondents found X years in perfect health equivalent to 10 years in impaired health, the utility of the impaired health state is calculated as X/10.

If a respondent preferred immediate death to 10 years in impaired health, they were offered the lead-time TTO. This method allows valuing health states considered worse than dead, which is what such a preference implies. This method described to respondents a life of 20 years in total, i.e. 10 years in perfect health followed by 10 years in the impaired health state considered worse than dead. Lead-time TTO involved the search for X years in full health (X <10 years) that respondents found equivalent to the 20 years described. As is usual, the utility of the impaired health state was calculated as (X−10)/10.

2.2 Decision Outcomes

The outcomes of TTO interviews were compared between both modes of administration for all health states combined and also for each health state separately in terms of: (1) central tendencies and variance and (2) clustering. Figure 2 shows the utilities for all health states combined and Fig. 3 shows the utilities for each health state separately.

Fig. 2

Time trade-off (TTO) utilities in personal TTO (n =36 observations for six health states) and tele-TTO interviews (n = 113 observations for six health states)

Fig. 3

Time trade-off (TTO) utilities per health state split by mode of administration (for the meaning of health states, see Appendix A)

2.2.1 Central Tendencies and Variance

Table 2 shows the mean and median utilities for each health state by mode of administration. For each health state, t-tests were used to determine if the mean utility was different in personal and tele-TTO interviews, and Wilcoxon tests were used to determine if median utilities were significantly different between both modes of administration. No such differences in means (all p-values >0.155) or medians (all p-values >0.164) were observed, with a similar lack of evidence found after utilities for all six health states were compiled for means (p = 0.78) and medians (p = 0.82), respectively. Next, to compare variance of TTO utilities between each health state, I performed Levene’s tests for equality of variance (which compare deviations from the mean) as well as Brown–Forsythe tests (which compare deviations from the median). No evidence for different variances is observed (all p-values > 0.07), with the exception of health state 35332, for which a significant result is observed with Levene’s test (p = 0.04). A similar lack of evidence exists for all health states compiled for both tests (both p-values > 0.48). Overall, there is little to no evidence of systematic differences in central tendencies or variance between modes of administration.

Table 2 Measures of central tendencies and variance for TTO utilities

2.2.2 Clustering Around − 1 and 1 and Other Modal Values

Norman et al. [26] compared online self-complete TTO with personal TTO and the main issue with TTO outcomes these authors identified was more pronounced clustering around certain utilities in self-completed TTO. In particular, the modal utility provided was 0 in their study, which was the earliest exit point in their TTO software. Figure 3 shows some evidence for clustering, as we can see notable peaks at 1.0 for states 11211 and, and −1.0 for states 24443 and 55555. This clustering may be explained by these health states being particularly mild or severe, respectively, and need not be problematic. More importantly, it should be determined if clustering (if it occurred) was affected by mode of administration. The following analysis was performed. First, for each health state separately, the modal utility is determined. Next, we determine by means of a Chi-squared test if the proportion of respondents that has utilities equal to that modal utility differed between modes of administration. The results can be found in Table 3.

Table 3 Modal utilities, proportion of respondents with utilities equal to that mode and Chi-squared tests for difference in distributions

Of note is that the mode for health state 22434 differs substantially between both personal and tele-TTO. Overall, we find that for two out of six health states, clustering around the mode is higher for personal TTO interviews and four out of six health states clustering around the mode is higher for tele-TTO. However, any differences observed are not significant. Finally, Fig. 2 shows that the overall distribution is clearly bimodal, with peaks at −1.0 and 1.0, hence I determined the proportion of at both peaks for both modes of administration. For personal TTO, 10.6% and 16.7% of the elicited utilities were 1.0 and −1.0, respectively. For tele-TTO, these percentages were 14% and 14.5%. That is, tele-TTO interviews elicited slightly more utilities equal to −1.0 and slightly fewer utilities equal to 1.0. A Chi-squared analysis showed this was not a significant difference, ${\chi }^{2}$(2, N = 894) = 1.70, p = 0.37. Hence, overall, there is no evidence for differences in clustering between both modes of administration.

2.3 Decision Quality

The quality of the TTO data is analysed by performing a set of analyses similar to those reported by Alava et al. [51] in their quality assurance programme for the EQ-5D-5L data in the UK. In addition to a set of analyses (similar to those) performed by these authors, additional analyses are included to test if violations of logical consistency were different between both modes of administration. If anything, these results point towards higher quality data in tele-TTO interviews, no differences were found when data quality was assessed as in Alava et al. [51] and the amount of problematic responses is lower for tele-TTO (Table 4).

Table 4 Data quality in personal time trade-off (TTO) and tele-TTO interviews, with quality indicators adapted from [51]