Introduction: step-wise outcomes and conditional effectiveness

Studies of vaccine effectiveness present typically relative risk reductions for various outcomes of interest for people with different vaccine exposures (e.g. vaccinated vs. unvaccinated or with vs. without booster doses). Many/most outcomes of interest typically have step-wise increasing (“hierarchical”) severity. E.g., for COVID-19, these outcomes may include any documented infection; symptomatic infection; hospitalization; severe disease; intensive care unit admission; and COVID-19 death [1]. Beyond vaccination, natural infection may also induce immunity and studies may evaluate the effectiveness (protection afforded by prior infection) for these outcomes [2]. Assessments of vaccination effectiveness are derived either from randomized trials [3] or observational studies [4, 5], and the latter are more numerous, especially when long-term outcomes are evaluated. Assessments of natural immunity effectiveness come only from observational studies.

Outcomes of hierarchically increasing severity may be subsets of each other. E.g., COVID-19 symptomatic infections are a subset of COVID-19 documented infections; and COVID-19 hospitalizations, severe disease events, and deaths are subsets of COVID-19 documented infection and of symptomatic infection. The subset pattern, however, may not apply perfectly for all the incremental severity steps. E.g., some deaths may be attributed to COVID-19 without testing documentation of infection but based on clinical picture; or, some deaths may happen without hospitalization. Here, we focus for simplicity on the situation where a more severe outcome occurs only in a subset of people who already had a less severe outcome. In principle, it is important to convey how much protection from the more severe outcome (e.g. death) is afforded by vaccination or natural infection for people who already had a less severe outcome (e.g. documented infection). This effectiveness estimate can be called conditional effectiveness, since it conditions effectiveness calculations on having the less severe outcome. Policy makers and public health communicators may be intuitively interested in conditional effectiveness. Instead of just reporting “how much is death risk decreased”, it is also informative to convey “how much is death risk decreased if someone does get detected with infection”. Here, a framework is presented for measuring, presenting, and interpreting conditional effectiveness.

Definitions and calculations

Let us assume that a vaccine (or natural infection) is found to have some effectiveness E(mild) for a mild outcome on a relative risk reduction scale. Additional protection against a more severe outcome among those who have already sustained the mild outcome may or may not exist. If there is no additional protection, then E(severe outcome) = E(mild outcome). E.g. suppose that the relative risk with a vaccine versus not being vaccinated is 0.2 for documented infection and 0.2 for death. Vaccine effectiveness is 1–0.2 = 0.8 for both outcomes. The vaccine reduces the risk of documented infection by 80% and the risk of death also by 80%. Therefore, among those who have had documented infection, the chances of dying are similar in both the vaccine and control groups. The vaccine protects from infection but, after this point, if someone is found to be infected the odds of dying are identical regardless of whether that person has been vaccinated or not.

Conversely, if additional protection if offered from vaccination or natural immunity against a severe outcome among those who already sustained the mild outcome, then E(severe outcome) > E(mild outcome). When vaccination or natural immunity paradoxically decreases protection for a severe outcome among those who sustained the mild outcome, it holds that E(severe outcome) < E(mild outcome).

Illustratively, if effectiveness is 90% against COVID-19 death, but only 80% against any documented infection: E(death) = 90% > 80% = E(documented infection). The vaccine decreases by 80% the risk of having a documented infection; if someone is nevertheless found to be infected, then the vaccine decreases death risk by 50%.

More generally, we can define the conditional effectiveness as the effectiveness among people who have already sustained an outcome that is less severe in a hierarchical scale of outcomes. Conditional effectiveness is measured after the less severe outcome has occurred.

By definition, effectiveness for an outcome is defined as:

$${\text{E}}({\text{less severe outcome}}) = 1 - {\text{RR}}({\text{less severe outcome}})$$
(1)
$${{\rm E}}({\text{more severe outcome}}) = 1 - {{\rm RR}}({\text{more severe outcome}})$$
(2)

and, by analogy, the same applies for a conditional outcome, i.e.:

$${\text{E}}({\text{more severe outcome}}|{\text{less severe outcome}}) = 1 - [{\text{RR}}({\text{more severe outcome}})/{\text{RR}}({\text{less severe outcome}})]$$
(3)

i.e.

$${\text{E}}({\text{more severe outcome}}|{\text{less severe outcome}}) = 1 - [[1 - {\text{E}}({\text{more severe outcome}})]/[(1 - {\text{E}}({\text{less severe outcome}})]]$$
(4)

where RR is relative risk.

If the total population in the compared groups (e.g. vaccine vs. no vaccine, booster vs. no booster, or previously infected vs. not previously infected) are provided along with the number of events for the different outcomes with hierarchical severity that are subsets of each other, then one can also calculate directly the conditional E(more severe outcome | less severe outcome) by using the events of the less severe outcome as the denominator and the events of the more severe outcome as the numerator. If person-time is involved in calculating incidence risk ratios and relative reductions in incidence rate, the concept is similar. To calculate the effectiveness for a more severe outcome conditional on a less severe outcome, one has to use in the denominator the number of person-time units of follow-up accrued after the occurrence of the less severe outcome.

Many studies perform time-to-event analyses (with or without proportional hazard models) to calculate vaccine effectiveness and its rate of decline over time, e.g. ref. [6]. Of note, when a participant has a less severe outcome and then develops a more severe one, these outcomes may happen with some time lag. When effectiveness estimates are presented for a given time period, less severe outcome events in some participants may be followed by more severe outcome events occurring at a later time point. E.g. when estimates of protection from infection and death are provided for 3 months of follow-up, some people infected at 3 months may die at a follow-up later than 3 months. However, when the typical time lag between the less and more severe outcomes is relatively short compared with the rate of decline of effectiveness, this is not a problem and one can still use Eq. 4 for a reasonable approximation of conditional effectiveness.

Nomogram

Figure 1 shows a nomogram for estimating the effectiveness for a severe outcome (e.g. death) conditional on a mild outcome (e.g. infection) as a function of the effectiveness for the severe outcome and for different values of the effectiveness for the mild outcome, E(i). With very high values of E(i), the conditional effectiveness is < 0, unless the effectiveness for the severe outcome is even higher than E(i). With very low values of E(i), the conditional effectiveness for the severe outcome is usually > 0, but, by definition, it is always less than the unconditional effectiveness for the severe outcome.

Fig. 1
figure 1

Nomogram of the conditional effectiveness for a serious outcome (e.g. death after infection) as a function of the unconditional effectiveness for that serious outcome for various levels of effectiveness for the mild outcome, E(i)

Full size image

Illustrative example

In a matched study of vaccination with three versus two doses [7], using events per person-years the short-term effectiveness was estimated to be 82% for documented infection, 85% for symptomatic infection, 87% for hospitalization, 89% for severe disease, and 84% for death. The effectiveness conditional on documented infection is 1 − [(1 − 0.85)/(1 − 0.82)] = 17% for symptomatic infection, 1 − [(1 − 0.87)/(1 − 0.82)] = 28% for hospitalization, 1 − [(1 − 0.89)/(1 − 0.82)] = 39% for severe disease, and 1 − [(1 − 0.84)/(1 − 0.82)] = 11% for death. The booster dose is very effective in decreasing the risk of infection, but it offers little additional advantage to survive among those who are detected to be infected.

Conditional effectiveness may change differently than effectiveness over time

In several studies, the observed decline of vaccine effectiveness over time varies for different outcomes [8,9,10,11,12,13]. For example, a study with 8+ months of follow-up after vaccination in North Carolina [8] found that there was considerable decline in protection against documented infection over time during 2021, but effectiveness for preventing death remained very high. Table 1 presents these effectiveness data for documented COVID-19 infection and for death and also shows the inferred effectiveness for death conditional on documented COVID-19 infection at 2, 5, and 8 months after vaccination. At 8 months, the Pfizer and Moderna vaccines protected only mininally from infection; however, among people who were detected to be infected, there was still substantial protection from death. Similarly, a study of CoronaVac in Brazil [9] found that > 180 days after the two-dose vaccination, effectiveness for documented infection had declined to 34.7%, but effectiveness for death was still 74.8%. Therefore, effectiveness for death conditional on documented infection was 61%.

Table 1 Effectiveness for documented COVID-19 infection and for death and conditional effectiveness for death in a study of vaccine use in North Carolina [8]
Full size table

Relationship of conditional effectiveness to case fatality rate and infection fatality rate

By definition, effectiveness against death conditional on documented infection, E(death | documented infection), represents the percentage reduction in case fatality rate (CFR). Similarly, effectiveness against death conditional on infection (documented or undocumented) represents the percentage reduction in infection fatality rate (IFR).

If R percent of infections happen in the vaccinated and 1 − R in the unvaccinated, then population IFR decreases by R*E(death | infection) compared with the pre-vaccination values, other things being equal. E.g., if 50% of infections happen in the vaccinated group and vaccination effectiveness for death conditional on infection is 80%, then population IFR decreases by 50% × 80% = 40%. More generally, a population may include people with different histories of previous infection(s) and vaccination(s). If Rk,m percent of infections happen in people with k status for previous infection(s) and m status for previous vaccination(s), and Ek,m(death | infection) is the effectiveness against death in that particular group, then population IFR decreases by the sum Σ(Rk,m*Ek,m(death | infection) for all k,m pairs. Diverse factors that influence fatality rates (Table 2) will affect the conditional effectiveness, if their changes over time differ in vaccinated versus unvaccinated people.

Table 2 Factors affecting the case fatality rate (CFR) (and/or the infection fatality rate (IFR)) in the vaccinated and in the unvaccinated and thus potentially also their ratio (the conditional vaccine effectiveness for death) if the factors change differently over time in the two groups
Full size table

Caveats and limitations

All effectiveness estimates in observational studies are subject to many biases inherent to epidemiological designs of vaccine effectiveness [14,15,16,17]. These biases also influence conditional effectiveness estimates. Moreover, confidence intervals (CIs) for vaccine effectiveness tend to be very large for the most severe outcomes, in particular death, unless very large populations are considered and many deaths are captured in analyses. CIs will be even wider for conditional effectiveness for death than for unconditional effectiveness for death. For standard formulas of variance (to calculate CIs) for relative risks and incidence rate ratios, see ref. [18].

Conditional effectiveness calculations are not causal estimates. This limitation applies, even if they were to be calculated from randomized trial data [19]. The literature on vaccine effectiveness for major outcomes (severe disease, death) would benefit enormously from large randomized trials of substantial follow-up and from systematic reviews of such trials. Such randomized trials and evidence syntheses would bypass many (but not all) of the problems of observational data. However, even such randomized trials would primarily answer in an unbiased way the question: how much benefit for severe disease or death is provided by the vaccination intervention (e.g. booster vs. no booster). For conditional effectiveness estimation, randomized trials may still be preferable to observational datasets, but accurate, unbiased conditional effectiveness may often still be difficult (if not impossible) to calculate since the two compared groups (vaccinated-and-infected and non-vaccinated-and-infected) are not the entire intention-to-treat randomized study population. These two groups may differ substantially by diverse selection forces. E.g. suppose that debilitated people (and thus more likely to die, if infected) are more likely to be infected despite vaccination and/or debilitated people are less likely to be infected among the non-vaccinated (e.g. if non-vaccinated debilitated people are extremely cautious). Then the calculated E(death | infection) may be low or even negative. Conversely, if very healthy people are more likely to be infected despite vaccination (e.g. if healthy vaccinated people feel very safe and get massively exposed), conditional effectiveness calculations may overestimate causal effects.

This difficulty to make causal claims is well-known in comparisons of randomized trials that involve post-randomization changes among participants, e.g. noncompliance, missing data, or post-randomization intermediate events [20,21,22]. In HIV vaccine trials a conditional outcome of interest is the reduction of viral load among those infected [23]. Some methods approximate causal effects by principal stratification, e.g. considering a group of participants who would be “always infected”, regardless of whether they are vaccinated or not [23]. However, for COVID-19 vaccines, the “always infected” group is difficult to define.

Personalized risk communication

Effectiveness is typically presented in relative risk reduction scales and thus this would be carried forward in conditional effectiveness calculations. However, for better communication and public health messaging, it is useful to use also absolute risk and risk reduction scales [24]. E.g. for E(death |infection), one can communicate the absolute risk reduction using the IFR estimates derived from seroprevalence studies. For example, for a community-dwelling person over 70 years old, IFR in high-income countries in 2.2% [25]. If conditional effectiveness E(death | infection) with vaccination is 80%, then one can convey that vaccination will decrease personal risk of death after infection by 2.2%*80% = 1.76%. With assumptions of 10%, 20%, or 30% chance of being infected in a given season, absolute benefit would then be 0.18%, 0.35%, and 0.53%, respectively. Numbers needed to treat can also be calculated accordingly [26].

Concluding comments

Conditional effectiveness can be readily calculated in studies of vaccine and natural immunity effectiveness. It may be used to communicate more appropriately the potential benefits of various vaccines, vaccination strategies and natural immunity under different circumstances. It should be made clear that these estimates are not causal and depend also on surrounding circumstances and features of the people who are infected. One should decide also whether circumstances and features of participants in a study are still relevant to current patients, a challenge of generalizability and of transportability [27].

Eventually, different outcomes carry different values and utilities for different people and in different settings that are considered in shared decision making [28,29,30]. Patients are primarily concerned with the outcomes that matter to them. Usually for COVID-19 vaccination, they would be mostly interested to know whether they die or develop severe disease and how vaccination might affect the chances of these outcomes. The concept of conditional effectiveness may be difficult to convey by physicians to patients, in the absence of better training of physicians in Bayesian risk communication and shared decision-making [31,32,33,34,35]. Still, the concept of conditional effectiveness may have more value for public policy considerations and public health messaging/communication than in time-compressed individual patient consultations.