Background

According to the mental health action plan for 2013–2020 by WHO, the provision of comprehensive, integrated mental health and social care services in community-based settings are essential [1]. If such services are to be affordable and accessible, all countries will have to prioritize when patients’ needs and demands for services exceed providers’ capacity of delivering them. Some patients will have to wait, while others, with lesser needs, may have to accept incomprehensive care. In Norway, referrals can be rejected if the reason for being referred does not fulfill priority-setting criteria [2]. One strategy to secure fairness and effectiveness in accessing affordable health service of good quality can be to apply priority-setting rules and grant patients with legal rights.

A healthcare system without waiting time requires enough capacity, which is probably too expensive to provide [3,4,5]. Even in the best-balanced healthcare systems, there is some waiting time. Since 1987, Norway has prioritized health services according to the principle of “severity of conditions”, where waiting time should reflect patients’ medical needs. In 1999, the Patients’ Rights Act introduced more specific measures such as waiting time targets and maximum waiting times. These were consistent with other nations such as Finland, Sweden, Denmark, UK and the Netherlands [6]. The Norwegian priority-setting policy has been revised several times [7]. To ensure horizontal equity in medical conditions, from institutional and geographical heterogeneity in praxis and capacity, national priority guidelines were specified for 30 different diagnostic groups in the years 2008–2009 [7]. These guidelines include eight psychiatric diagnoses and 22 somatic diagnoses. For example, patients with severe and moderate depression are recommended to wait no longer than 2 weeks and 8 weeks, respectively. The overall national waiting time target for mental health services in 2021, is 40 days for adults, 35 days for children and adolescents, and 30 days for alcohol and substance dependencies. Detailed description of the Norwegian white papers on priority setting are presented elsewhere [8].

Despite the existence of the Patients’ Rights Act and national generic priority setting guidelines, few studies have examined the performance of these regulations, when it comes to implementing the “severity of conditions” principle. From a Norwegian national-level study, it was reported that although waiting times were reduced after maximum waiting targets were enforced, patients with severe medical conditions were given low-priority [9, 10]. They also reported that high-priority patients had higher probability of waiting excessively. A Norwegian regional-level analysis, found no direct association between actual waiting time and patients’ priority status [11]. They also speculated that implementing overall waiting time guarantees may have pushed the actual waiting time closer to the maximum. Due to the lack of evidence in mental health services, there is a need to investigate whether the “severity of conditions” principle for people with mental illness has worked adequately under the policy.

In Norway, as in many countries, a referral from a general practitioner (GP) is necessary to access specialized healthcare. Several studies report that the specialists who received the referral letter often finds the information insufficient to assess the condition and prioritize the urgency [12,13,14,15,16,17,18]. Therefore, disagreements between specialists within and across providing institutions, on what patients’ needs are and who will benefit from specialized treatment, can lead to inequalities in access [19, 20].

Evidence suggests that waiting time is associated with patient outcome [21] and satisfaction [22, 23]. Targeting patients with the greatest needs and letting them have shorter wait-times is at the core of priority setting for elective treatment, similar to triage in acute care. A healthcare system that allows specialists to exercise discretion when assessing individual patients’ needs, must expect substantial variation. A supplement strategy could be to introduce the patients’ subjective opinions on their needs, without taking away the specialist’s ability to make the decisions. Such a strategy appears to be time and resource consuming. However, with the introduction of Patient-Reported Outcomes Measurement Information Systems (PROM-IS) in healthcare, clinicians and clinical managers have the opportunity to include the patient’s self-assessment in the decision-making process and planning of care. Pioneers of PROM-IS, e.g., New England Baptist Hospital (NEBH), have invested substantially in strategies that aim to provide patients with greater influence over their own care. For instance, based on Patient-Reported Outcomes Measurement (PROM) collection at NEBH, a predictive AI model [24] measures potential risks before treatment and is used to share decision making between patients and providers.

Studies of referral texts and specialists’ clinical assessment in electronic medical records (EMR) require extensive resources and patient consent. The Lovisenberg electronic Patient Reported Outcome Measurement (LOVePROM) project, collects PROM as part of a quality register at a community mental health center (CMHC) in Oslo. The project was able to provide an anonymous dataset of administrative data from the electronic medical records, with waiting time, codes from the 10th revision of the International Statistical Classification of Diseases and Related Health Problems (ICD-10) and PROM, episodes, and procedures. In this case, the content of the referral letter is unknown. Without the GPs’ description of urgency and need, historic waiting times should work as a proxy to urgency and needs, as perceived by the clinicians at Lovisenberg hospital.

To create groups of patients, this study investigates ICD-10 diagnosis, particularly regarding severity in depression. As noted, Norwegian clinical guidelines have specific recommendations for waiting time in depression. To examine individual needs, we use PROM collected just before the onset of therapy, specifically data from the Clinical Outcomes in Routine Evaluation-Outcome Measure (CORE-OM).

Aim of the study

This study investigates if patients’ prioritization on waiting time is consistent with the principle of “severity of conditions”, where waiting time should reflect patients’ medical urgency within the given capacity. Specifically, we examine the association between patients’ diagnoses and symptom severity at the start of treatment and their waiting time. If the system works well, we hypothesize that patients reporting severe distress have had significantly shorter wait-times, compared with low and moderately distressed groups measured by CORE-OM. We also hypothesize that severely depressed patients have had shorter wait-times by several weeks than patients with moderate to mild depression. The findings will be discussed in relation to when pre-PROM should be collected— together with referral assessment or at onset of treatment after waiting time. This quality improvement study is significant as the findings will help highlight if there are good reasons to recommend a shift from PROM collection after waiting time is over, to before waiting time is set.

Method

Materials

This study analyzed routinely collected data from EMR and PROMs at Lovisenberg Diaconal Hospital CMHC in Oslo. The CMHC covers a catchment area of approximately 200,000 inhabitants in Oslo’s inner city, with five clinics providing services for mental disorders, addiction, and substance use disorders. One of these clinics is defined as a return-to-work program, where also mild conditions could be prioritized to prevent workers from sick leave. We included new patients referred to specialized mental health outpatient programs between January 2017 and August 2020.

The data set was extracted from Lovisenberg CMHC’s quality register (LOVePROM) in an anonymous form. Using CORE-OM, each patient’s health status was assessed three days before the first consultation. The CORE-OM (Norwegian versionFootnote 1) is a 34-item global distress measure with four subscales: subjective well-being, problems/symptoms, life functioning, and risk/harm [25]. Patients’ final diagnoses were received by the EMR at the end of treatment. Figure 1 summarizes the data collection time-points and the main information utilized.

Fig. 1
figure 1

Timeline for referral, waiting time and treatment (with data collection timepoints)

Full size image

Variables

Table 1 presents a list of the variables used. Waiting time was defined as the time between the referral from the GP and the start of treatment (Fig. 1) and was measured in weeks. In our regression analyses, waiting time was the dependent variable while the other variables were independent.

The health need characteristics were collected at the start of treatment using the CORE-OM questionnaire. The CORE-OM total score was based on all 34 questions in the instrument. The score is defined on an ordinal scale ranging from 0 to 4 for each question and 0–136 for the 34 questions in total. We used the mean score for all questions multiplied with 10, so that CORE-OM total score ranges between 0 and 40. In parts of our statistical analysis, we categorized the CORE-OM total score into five CORE-OM severity categories [26, 27]: Under clinical cut-off (0–10); Mild (10-15); Moderate (15-20); Moderate severe (20-25); and Severe (25–40).

The dichotomous variable of being at “risk-to-others” was based on two risk questions (Q6 and Q22) from the CORE-OM questionnaire, confer Supplementary Table 1, Additional File [25, 26, 28]. If a participant responded to at least one of the questions that was above a given trigger level, the risk variable was assigned the value 1, otherwise the risk variable took the value 0 (i.e., if the responses to both risk questions were below the trigger levels). The variable of being at “risk-to-self” was similarly defined based on a set of four risk questions (see Supplementary Table 1, Additional File).

The diagnosis variables (ICD-10) were recorded at the last episode of treatment for the main categories: Mental and behavioral disorders (F-chapter), Symptoms (R-chapter), and Factors influencing health status (Z-chapter). For patients with depression, we also obtained variables for eight diagnostic subcategories.

Data analysis

The original sample consisted of 6314 patient records. After excluding 206 incomplete records, the net sample consisted of 6108 patients. From this sample, we defined a subsample consisting of 1583 patients with depression.

We estimated basic descriptive statistics for the included variables— means and standard deviation for the continuous variables, and sample proportions for the dichotomous variables.

For patients with depression, the national guidelines advise maximum waiting times by level of severity (2 weeks for severe depression; 8 weeks for moderate depression). For our subsample of patients with depression, we constructed box plots for the observed waiting time by the severity level. This enabled us to compare the distribution of waiting times with the national guidelines.

For mental health patients in general, the national political target for the average waiting time was to be under 40 days in the year 2020 (approximately 6 weeks). Except for depression, as mentioned, the guidelines do not state maximum waiting times depending on ICD codes. To assess the association between waiting time and severity measured at the start of the treatment, we constructed box plots for the waiting time based on CORE-OM severity categories.

Waiting times can potentially be associated by any of our included variables and the associations can potentially confound each other. To investigate this, we estimated four linear regression models (Model I–IV) for the sample of patients with depression, using waiting time as the dependent variable and different sets of independent variables.

In Model I, we included gender and age – variables that were available at the time of referral. In Models II and III, we added the health status characteristics (CORE-OM available at the start of treatment) and the depression diagnosis variables (available at the end of treatment), respectively. Model IV included both health status characteristics and the depression diagnosis variables. It can be used to assess whether these two sets of variables confound each other or compete for explanatory power. Explanatory power was measured by the adjusted coefficient of determination.

We then estimated corresponding regression models for the full sample consisting of all patients (Models V–VIII). The difference between Models I–IV and these models is that the diagnosis variables in the latter were based on ICD-10 main categories, while in the former, were based on ICD-10 (depression) subcategories. We used Stata 16.0 to perform the analyses and a 10% significance level.

Results

Table 1 shows the descriptive statistics for the sample and variables included in the regression models. Our study sample was predominantly female, 20–39 years old, and diagnosed with Mood [affective] disorders, Neurotic, stress-related, and somatoform disorders. Patients with depression (n = 1583) accounted for 25.8% of the whole sample. The mean CORE-OM score was 17.4, with a standard deviation of 6.0 scores, that is, over the clinical cut-off being 10. The distribution of patients according to CORE-OM categories are reported in Supplementary Table 2, Additional File. Very few patients (1.1%) reported being of risk to others, while a much larger group (19.0%) reported being at risk of self-harm. During the study period, patients in the study sample waited, on average, 8.1 weeks before treatment, varied by 4.9 weeks (Table 1).

Table 1 Descriptive statistics for the sample used in the regression analyses (N = 6108)a)
Full size table

The results of the prioritization performance are presented using boxplots in Figs. 2 and 3. In the sample, waiting time varied from a median of 7 weeks for patients with severe and moderate depression, to 9 weeks for those with mild depression (p = 0.000). The boxplot depicts overlap among the three severity levels and variation within each level. The results also demonstrated that waiting times substantially exceeded national guideline recommendations. Patients with moderate depression (32.4%) and severe depression (83.3%) waited longer than 8 weeks and 2 weeks, respectively (Fig. 2).

Fig. 2
figure 2

Waiting time for patients with depressive disorders, by depression diagnosis (N = 1583)

Note: The distribution of waiting time for three categories of depression were significantly different (Kruskal-Wallis test with p-value < 0.001). The lower and upper dashed lines represent the Norwegian national guidelines’ maximum waiting time for severe depressive disorder (2 weeks) and moderate depressive disorder (8 weeks), respectively [29]

Full size image

The results for overall diagnoses also showed the waiting time exceeding the national prioritization target (dashed line: 6 weeks). The variance narrows as the severity increases, while medium waiting times do not indicate a steady decrease. Specifically, patients with moderate to severe conditions waited seven weeks, while patients with mild conditions or below clinical cutoff waited 8 weeks. The difference was significant but small (Cohen’s d = 0.198, p = 0.069)  (Fig. 3).

Fig. 3
figure 3

Waiting time by CORE-OM severity category (N = 6108)

Note: The dashed line at approximately 6 weeks (40 days) represents the Norwegian national target for average waiting time for mental health outpatients [8]

Full size image

The multivariate regression models used information obtained at different time points to elaborate potential factors that correlated with waiting time (see Table 2). Model I shows that patients over 60-year-old wait 1.8 weeks longer than those aged 20–29 years. Longer wait-times in this group can be found after including more information in Model II and Model III.

Model II adds on the variables’ CORE-OM scores, risk-to-others, and risk-to-self. As CORE-OM scores rise by one point, the coefficient of CORE-OM denotes the difference in waiting time. In Model II, the estimated value is -0.069, which means that an increase of one unit on the CORE-OM would reduce waiting time by less than one day. It shows a significant, but small impact (Cohen’s d = 0.141)Footnote 2. The coefficients for risk-to-self and risk-to-others denote the change in waiting time when comparing patients’ CORE-OM risk subgroups above and below the risk-alert line. The significant estimate for risk-to-self was -0.906 (p < 0.01) for Model II and -0.844 (p < 0.01) for Model IV, showing that the patients with the risk-to-self alert waited around 1-week less than those without.

Models III and IV show regressions that include a depression diagnosis, while Model III excludes CORE-OM. Compared with patients with mild depression, patients with moderate and severe depression have shorter wait-times (Model III & Model IV).

Measured by the magnitude, with PROM estimates in Model IV, the moderately and severely depressed patients had shorter wait-times by 1.3 weeks (Cohen’s d = 0.26) and 1.5 weeks (Cohen’s d = 0.29), respectively. This magnitude is slightly smaller than estimates without PROM, which are shorter by 1.5 and 2.1 weeks (Model III).

Results from Model II and Model IV show that CORE-OM scores and depression stages are competing for explanation power. The change of CORE-OM scores is consistent with the change in depression stages. As we included the depression diagnosis, the CORE-OM scores decreased from significant (coefficient = -0.069 in Model II) to insignificant (coefficient = -0.027 in Model IV). The unit change from moderate to severe depression yields an increase of 4.8 CORE-OM scores.

Table 2 Multivariate regression models for waiting time. Patients with depressive disorders (N = 1583)a
Full size table

Diagnoses do not directly describe the actual severity of conditions. Therefore, we used PROM as an expression of patient severity. Table 3 reports the waiting time and influential factors for all patients with ICD-10 diagnosis. Compared with the reference group (aged 20–29), the youngest patients (age < 20) had significantly shorter wait-times— nearly one week less (-0.917; Model V). After controlling for severity in CORE-OM scores in Model VI, we observed that older patients (aged 40–49 and 50–59) have somewhat shorter wait-times.

Table 3 Multivariate regression models for waiting time. Coefficients and standard errors (N = 6108)a
Full size table

Compared with patients without a risk-to-self alert, patients with an alert had a shorter wait-time by 0.7 weeks in Model VI and by 0.5 weeks in Model VIII. The coefficient for CORE-OM scores in Model VIII indicates that patients with a one-unit increase in CORE-OM had nearly 0 (0.023 ; Model VIII; p < 0.1)-week shorter waiting time. Model VII was analyzed without PROM. Comparing with PROM in Model VIII, the increase of adjusted R squared changed from 3.9 to 4.2%. Further, patients in different diagnostic groups had systematically different waiting times. All other diagnostic groups waited longer than the reference group F20–F29 (Schizophrenia, schizotypal and delusional disorders). The effect was largest for patients diagnosed with F90–F99, waiting 6.5 and 6.6 weeks longer than the reference group with or without PROM data (Model VIII & Model VII) (Table 3).

Discussion

In this exploratory study, we found that diagnoses and symptom severity, described by PROM at the start of treatment, were associated with waiting time. However, the associations were smaller than expected, and many patients waited longer than the recommendations of the official guidelines.

In this study, the overall mean waiting time was eight weeks, two weeks longer than the national target in 2021. This could be because of limited capacity in the services. Compared with patients with schizophrenia, other diagnostic groups had longer wait-times. This is consistent with clinical recommendations. Relatively few patients were diagnosed with an unspecific diagnosis (R00–R99 and Z00–Z99). This indicates that patients with mental health issues primarily requiring specialized treatment were accepted into the health service. However, a more specific referral evaluation is still required, as indicated by a relatively high proportion of patients with mild depression diagnosis. We found a significant association with shorter waiting time for the youngest patients, but it was less than a week.

The patients reported a mean CORE-OM total score of 17.4, with a standard deviation of 6.0. This indicates that the CMHC prioritize patients that are over the clinical cut-off of 10. In this respect, referral letters seem to inform the priority setting well. There was an association between higher total scores on CORE-OM and shorter waiting times, but this association was weak. A ten-point higher score on CORE-OM, which is a clinically relevant size, was associated with only 0.2 weeks, i.e., one-two days shorter waiting time (Table 3, Model VIII).

The most prominent finding of our study is that severely depressed patients wait much longer than recommended. Nearly all patients diagnosed with severe depression waited longer than the recommended two weeks. This might indicate a shortcoming in referrals if they do not identify the most severe cases within the most frequent mental disorder. Most patients reporting moderate-severe and severe levels of distress on the generic instrument CORE-OM waited five to ten-eleven weeks. An alternative explanation is that some of these patients’ condition became worse during the relatively long waiting time.

Patients with depressive disorders, reporting risk to self-harm and patients with moderate to severe depression had a shorter wait-time by less than two weeks, than patients without self-harm alert and mild depression, when gender, age, and CORE-OM scores were adjusted for (Table 2, Mode IV). This is consistent with our observations (Fig. 2) where patients with generally higher symptom severity waited almost as long as patients with milder symptoms, possibly indicating failure of referrals to identify more severe conditions. Compared to diagnosis, PROM identifies patients being at risk-to-self and providing them with shorter waiting times (Table 2, Model II, IV & Table 3, Model VI, VIII).

The assessment of risk is a top priority within routine counselling and psychotherapy services. In recent years, staff have received training in this area. The risk domain of CORE-OM, alert clinicians to patient’s risk of harm to self and others. Responses on risk domains do not have a high predictive value. However, the alert gives providers an opportunity to consider treatment actions to relieve distress and potentially prevent harm, e.g., shorter waiting time. Research suggests that differences between practitioner-rated and client self-report assessments are to be expected and has indicated that the rates of difference can be relatively high (i.e., > 50%). Bewick and colleagues found that the CORE-OM risk domain identified 44% of clients as “at risk” while the practitioner assessment identified 10% of clients as being “at risk” [30]. For the overall sample in their study (n = 25338), 18% of clients were classified by the practitioner as presenting no risk when the CORE-OM risk domain identified them at risk [30].

In our study, relatively few (1%) patients reported a violence alert and a relatively large group of patients (19%) reported elevated risk of harm to self. Their wait-times were similar (risk-to-others) or were shorter by less than one week (risk-to-self), compared with patients reporting under the thresholds for these risk behaviors. This observation could be viewed as consistent with Bewick and colleagues reporting that clients and practitioners have different views of risk. The difference in the size of patients identified at risk— 44% reported by their study and 19% reported in our study— can result from different patient selections or thresholds [30].

The most comprehensive model in our study including all factors (Table 3, model VIII), explained only 4.2% of the variation in waiting time. This indicates that factors other than diagnoses and symptom severity at onset of treatment, explain most of the variation in waiting time. These factors may be: long waiting time may have worsened the condition from time of referral to onset of treatment; information on risk behavior might not have been described by GP’s, and therefore, not available when referrals were assessed; GP’s may not have asked patients risk questions, or patients may not have trusted their GPs with their honest answers. Lack of trust in GPs and lack of openness with their current mental health state, could be one reason why patients ask for a referral to specialized mental health services. Lack of competence or time to deal with mental health issues could be GPs’ reason to refer. Moreover, clinicians evaluating referrals may give weight to factors other than condition severity, for example, information about earlier treatment, motivation, or psychosocial factors.

Variation in waiting time normally expresses prioritization consistent with the principles of horizontal and vertical equity. The World Health Organization in the year 2000 advocates concepts of horizontal equity— individuals or groups with the same level of need are treated equally [31]. Vertical equity relates to cases where people with unequal needs are treated proportionately differently, preferentially providing to those with the greatest need.

Previous studies have found that quality of referrals and other forms of communications in transition of care can affect patient’s safety, e.g., when urgency is not described or inadequately understood by providers in the pathway [32]. Holman and colleagues described how disagreement on referral assessment may lead to random priority setting effects in a study setting [20]. Twenty anonymous case vignettes referrals were classified by 42 admission team members at 16 CMHCs. The same “patient” was assessed as in urgent need or no need at all, suggesting low degree of interrater reliability. A way of improving quality in assessment of referrals could be to have all referred patients undergo an evaluation by a specialist as part of the priority setting process.

A more cost-efficient alternative could be to collect PROM with a validated measurement at the time of referral, allowing the patients’ own descriptions of health condition and symptom severity to inform the intake process.

Recommendations and limitations

To our knowledge, PROM is not routinely used anywhere to accompany the assessment of referral letters for priority-setting decisions. We suggest that this is implemented, and that further research investigates possible effects on priority setting and waiting time compared to guidelines.

There are also some limitations in the setting. First, there is inconformity of data collection timepoints. Ideally, we would wish to know patients’ subjective health status when the referral letter is evaluated and waiting time is set. In this study, such data was not available to inform decision making on waiting time together with referral assessment. Therefore, we are not looking for agreement, as for inter-rater reliability, between specialists at CMHC and patients perceived the urgency. They did not share the same information at the time. Second, the GPs problem description (International Classification of Primary Care codes) and ICD-10 codes are not the same, and patient’s condition described by PROM may well have changed during waiting time and treatment, resulting in a different ICD-10 classification compared with the first consultation. We only have ICD-10 codes and find that the last code best represents the diagnostic conclusion. This is a limitation of the study setting. Our aim was to determine whether the current practice of referral assessment works well in accordance with the priority setting principal of “severity of the condition”. Since PROM is not routinely used anywhere to accompany the assessment of referral letters for priority-setting decisions, we have not yet been able to perform a naturalistic study. With LOVePROM or other electronic infrastructure for administering questionnaires to patients, it could be feasible to assess CORE-OM from patients together with referral letter within days of the referral. Third, the risk-to-self and risk-to-others items are dependent on our clinical therapeutic exercise, which may or may not apply to other clinical settings in mental health clinics.

Conclusions

Our results have direct implications for the waiting time target policy, and routine collected outcome measures. As expected, this study reports variation in waiting time between groups of patients at a CMHC. Overall, the patients wait in accordance with the severity of condition principle, i.e., higher urgency leads to shorter waiting times. However, the trend is not strong. Therefore, we advocate that there is substantial room for quality improvements in priority setting on waiting time. The role of managing access to specialized health services should aim to target patients with high burden of disease and with risk of reduced outcomes if waiting time is too long [11, 33, 34].

Without validated rating instruments and checklists, the room to exercise clinical discretion is large. PROM, in this case CORE-OM, has some explanatory power. We suggest it could be substantially stronger if PROM could inform decision makers before setting waiting time.

Based on these examples and our observations, we argue that waiting time targets, for groups of patients, not only depend on capacity. Such policy’s efficiency, i.e., the principle of severity of condition or the principles of horizontal and vertical equity, depend on an agreement on which patients belongs in the prioritized groups. Therefore, we suggest that future studies should investigate if routine collection of pre-PROM, assessed together with referral letters, can better inform specialists when deciding on waiting time. We expect to find that such a reform will improve equity, so that patients with higher distress and risk of lost prognosis, have shorter wait-times. We also expect to find less variation in waiting time within groups of similar conditions. We encourage the LOVePROM project to start collecting pre-PROM and evaluate the effect on priority setting.