Background

Over the course of the last few years the design and analysis of clinical trials have come under scrutiny mainly because of the increasing number of new treatments approved coupled with the need to understand their actual clinical and economic impact.

We have noted an increasing number of registrations for a relatively limited number of costly drugs/classes of drugs for a broad range of different human tumors. Therefore, a potential discrepancy has arisen in that there is a risk that the preclinical rationale and the actual clinical benefit/economic ratio leading to these registrations have not been sufficiently scrutinized.

In this regard we and others have cautioned that given the increasing cost of these drugs together with their safety profile it is recommended their approval in clinical practice should only be granted only when they show a proven impact both clinically and economically [1,2,3].

Despite several red flags that have been raised (e.g. [4]), FDA approved treatments have boomed during the last year which may lead to significantly increased costs for the health systems worldwide.

There is no doubt that such treatments in this era of precision oncology have the potential to greatly improve the clinical outcomes of distinct subsets of patients [5]. We believe there are limitations to the current approvals with respect to the following: (a) the vast majority of these approvals appear to have been inspired by a “one-size fits-all” approach rather to a patient / disease tailored treatment [6]; (b) the pricing of these new targeted therapies did not differ regardless of if they achieved a surrogate end point instead of an actual gain of survival [7]; and (c) we believe that many of the trials upon which these approvals were granted lack an appropriate control arm. For example, in 2019–2020, six cancer drugs have been approved by the FDA with no apparent control arm (Table 1), and in one instance despite an advisory panel’s concerns about the drug’s toxicity and the lack of randomized clinical data [4].

Table 1 FDA approvals in 2019–2020 for precision drugs with no control arm
Full size table

As such we believe that there is a broad consensus for the need to have unbiased patient selection (in particular with regard to age, PS, staging and including an optimal control arm) to have the necessary rigorous control for a clinical trial. Unfortunately, it is our belief that many of these simple principles are far from being homogenously applied [17, 18].

The “Mesothelioma model”

Immune checkpoint inhibitors

A “one-size-fits-all” approach becomes even more accepted when rare/ hard-to-treat cancers are considered as few resources are generally allocated to these cancers.

Malignant Pleural Mesothelioma (MPM), is one such rare/hard-to-treat cancer, an aggressive occupational cancer with heavy social impact, and we shall use MPM in the following sections to represent an “ideal” model to study this potential issue.

Currently we are seeing the rapid identification of potential targets and emergence of multiple novel therapies that are tested in Phase II trials for this neoplasm. All are welcome as potential as they may prove to be key turning points for the treatment of this stubborn neoplasm. On October 2, 2020 the FDA approved the use of the immune checkpoint inhibitors Ipilimumab/Nivolumab as a first-line treatment for adult patients with unresectable MPM [19], on the basis of the results from the CHECKMATE-743 (CM-743) clinical trial [20, 21].

Whilst this approval has been warmly welcomed by the clinical community as it is essentially the first major approval of a treatment of MPM in the firstline setting since the initial approval of cisplatin/pemetrexed in 2004 [22], concerns regarding the solidity and reliability of the results obtained with Immune Check-point inhibitors (ICIs) for MPM have been raised [23,24,25,26]. Indeed, using innovative statistical tools that calculate either Survival Inferred Fragility Index [SIFI], or the Restricted Mean Survival Time Difference [RMST-D] we and others have described limitations to various standard therapies for other clinical settings including ICI [27,28,29,30].

These first results, have prompted us to examine in more depth the three-year results of CM-743 and analyzed the Survival-Inferred Fragility Index (SIFI) including an additional censoring analysis of the updated three-year results of CM-743 trial [21] with the same methodology used in our previous analyses [31]. Our analysis raises some important issues. In particular we would argue that the methodology used for OS analysis and the subsequent conclusions are still associated with informative censoring [32], due to differential censoring favoring the control arm (p = 0.026) [20, 21]. It is noteworthy that, after performing a sensitivity analysis accounting for these censoring imbalances, we observe that the results were no longer significant (HR 0.85, 95%CI 0.71–1.02; p = 0.089).

Using the Survival-Inferred Fragility Index (SIFI) method with the three- year outcomes (21), we found that the SIFI was 6 patients, representing only 0.99% of the total sample size. This finding indicates that a small variation in the study population could in effect overturn the conclusions of the study, and suggest that the original trial data also lack of statistical robustness.

In this regard, we also report that the OS curves of the interventions for the MAPS trial (33) and the updated CM-743 trial still overlap for both intervention and control (Fig. 1). It must be noted however, that such a comparison cannot be considered to be as reliable as a head-to-head comparison in a randomized control trial, nevertheless we believe it is worthy of careful consideration.

Fig. 1
figure 1

Curves show virtual comparisons between the reconstructed overall survival curves of control and intervention groups for the updated CheckMate743 and MAPS studies. HR indicates hazard ratio

Full size image

The long-term 3-year follow up data for CM-743 suggests that only non-epithelioid MPM appears to derive any benefit from Ipilimumab/Nivolumab. This by itself is a noteworthy result, as this subtype of MPM is traditionally associated with a worse outcome, and poor response to cisplatin/pemetrexed based therapy. The results of three-year-analysis of CM-743, might therefore benefit non-epithelioid histological subtype because they seem to be associated with a significant effects size.

However, we found that even in this subset severe limitations exist and, in particular, the original significant differential censoring for this subtype still represents a relevant unresolved issue that introduces a pivotal bias to the results. Alternatively, it is also possible that the observed results in the intention to treat subjects are driven by a genuine effect in the non-epithelioid subset and were masked by the lack of it in the epithelioid subtype.

There also remains the possibility that this potentially significant finding, as it may be underpowered, as underpowered studies provided with the flexibility of the exploratory analysis often produce false positive or exaggerated results [34].

Whilst compelling, we propose that the results from this subgroup analysis should be considered as hypothesis generating, because the CM-743 trial was neither designed nor powered to answer this important question. It is our belief that additional prospective and randomized trials will be required especially for the non-epithelioid -subtype, to fully establish the impact of this new therapeutic option for patients with MPM. to establish its role as a “front-line” treatment for MPM.

As far the Phase II trials of ICI published so far, the use of suboptimal control arms [35], the widespread use of surrogate primary end points [35, 36], and the concerns about patient selection (with respect to performance status 0 (PS0), young age, and very early stage) suggest that we need to revisit clinical trial design in MPM. In this regard the Phase II trial combining first line ICIs with chemotherapy [37], provides an example of this issue in that efficacy is compared to “historic” response, and roughly 50% of the study population at PS0 (ECOG PS0 of 41.8%). The validity of the benefits observed in this trial have subsequently been questioned following independent re-analysis [24].

Other authors have presumed the efficacy of Nivolumab in second line on the basis of only 34 MPM patients but not considering the very broad range of actual responses and even the broader standard deviation of the results (Fig. 2; [38]).

Fig. 2
figure 2

Comparison of OS achieved by the latest trial for MPM

Full size image

New targeted therapies

The issues with regard to inappropriate trial design/methodology are not restricted to those of just ICI. Lack of or suboptimal control arms have been raised frequently as an issue in several of the recent Phase II trials of new agents/therapies in MPM [39,40,41].

The current problem with biomarkers

Many of the current clinical trials conducted in MPM have utilized a basket case approach, often involving the use of candidate biomarkers [42]. However, in many instances the pre-clinical data predicating these biomarkers may be flawed. In the following sections we shall discuss in more depth some additional examples of Phase II trials that raise concern. All center on a gene whose expression is frequently inactivated/lost in MPM, BRCA1-associated Protein 1 (BAP1) [43], or to BRCA1 itself.

It was first suggested that BAP1 inactivation/loss was associated with sensitivity to enhancer of zeste 2 (EZH2) inhibitors in MPM (44). However, a Phase II trial of an EZH2 inhibitor (Tezemetostat) has been completed in MPM [45]. In this trial the authors state that despite stratifying the responders by BAP1 status no statistically significant differences (p = 0.264) in survival were observed indicating BAP1 mutational status has no influence on response to Tezemetostat [45].

In a similar manner, BAP1 inactivation/mutation was linked to sensitivity to Poly(ADP-Ribose) Polymerase (PARP) inhibitors [46, 47]. Two Phase II clinical trials of PARPi in MPM have now been conducted which have assessed whether BAP1 status plays a role in sensitivity/response [48, 49]. In both trials, overall it can be concluded that PARPi have limited activity in MPM including patients with BAP1 mutations, further confirmed by an independent study [50], and the rationale for the supposed efficacy of PARP1 inhibitors in MPM patients bearing BAP1 mutation remains inconclusive [48, 51, 52], and warrants further attention.

BRCA1, BAP1 and vinorelbine

A potential role for BRCA1 as a biomarker for sensitivity to vinorelbine was identified in 2012 [53], and potentially confirmed through a pooled analysis [54]. On the basis of this observation a Phase II clinical trial (NCT02139904) was conducted with patients randomized 2:1 to receive either active symptom control with oral vinorelbine versus active symptom control (ASC) every 3 weeks until disease progression, unacceptable toxicity or withdrawal [55], and whilst the trial met its stated primary goal with respect to improved PFS, BRCA1 did not predict resistance to ASC + vinorelbine [55]. This appears to confirm the earlier observation by others that BRCA1 was not a good biomarker for stratifying sensitivity to vinorelbine/cisplatin treatment in MPM [56].

Intriguingly, an analysis of some of the patients treated with vinorelbine demonstrated that loss of BRCA1 or that of a separate new biomarker MAD2L1, a gene transcriptionally regulated by BRCA1 [57]. Moreover, a retrospective analysis of the MS01 trial (NCT00075699), found a small, though non-significant, overall survival disadvantage associated with BAP1 expression in tumors from patients treated with vinorelbine (58). These results suggest that a more comprehensive Phase III biomarker driven trial is warranted in order to truly determine the potential utility of BAP1, BRCA1 and MAD2L1 as biomarkers for sensitivity to vinorelbine/cisplatin combinations.

Returning to PARPi, it must be noted that a Phase II study examining the combination of Niraparib plus Dostarlimab is currently being conducted (NCT04940637) in NCSLC and MPM [59], for patients with confirmed positivity for germline or somatic homologous recombination deficient (HRD) status and tumor PD-L1 expression (tumor proportion score 1%) and must have experienced disease progression or recurrence during or after at least 1 systemic therapy for advanced metastatic disease. Unfortunately, the study suffers from some of the weaknesses that we have already raised for other trial methodologies such as ECOG status 0 and is limited by being a single-arm prospective study, with the outcome design generated on assumptions for PFS based on historical data.

This problem is compounded by the number of new second-line treatments tested so far, as none appear to top the survival of the drugs currently used in the same settings (Fig. 2) and that in general ICIs for MPM do not show any true superiority compared to standard treatments within real world settings [23, 60], a matter that we have raised with respect to the results of several other trials for MPM recently published [35, 39, 40].

The financial impact: are we bearing the brunt?

Previously in 2011, the Oncology Commission of Lancet stated: “. . the cancer profession and industry should take responsibility and not accept a sub-standard evidence base and an ethos of very small benefit at whatever cost; rather, we need delivery of fair process and a real value from new technologies”, to achieve reliable and quantitative evaluations of health outcomes and costs, for both equity and affordability [61].

Since then, numerous attempts have been made to draw attention to the medical and scientific communities that, whilst these new developments in cancer treatment have emerged, with outcomes that have benefited patients justifying their high costs costs, currently, this benefit/cost ratio has progressively reversed up to a break-even point and we are now approaching a situation of unaffordability of cancer care, and to our mind as a priority, this trend imposes a challenge regarding the ethics of affordability for future cancer treatment. (Fig. 3) [62]. Our results fit with recent analyses which have determined that in particular, ICI combinations such as Ipi/Nivo exceeds the willingness-to-pay threshold from the perspective of US payers for the treatment of MPM [63,64,65]., which place the costing for the incremental cost-effectiveness ratio (ICER) of $372,414.28/QALY [64] and ICER of $475,677/QALY [63] for the newly approved ICI combination in MPM respectively. In this regard, the Phase III CONFIRM trial [66] has been cost-effectiveness built into its trial design and should provide some important data in this regard.

Fig. 3
figure 3

The rate of growth in costs (red) vs. outcomes (green) of cancer care, is here expressed as the derivative D[fx0] of both functions (i.e.: the angular coefficient at any specific point of each field)

Full size image

How can we improve on this?

To put this into perspective, a recent analysis suggested that the approximate cost for a 24-week treatment of vinorelbine is $515, while that for six cycles of gemcitabine is approximately $887.76 [67], and the cost-effectiveness for the standard cisplatin/pemetrexed using QALY in the UK was ranges from £20,475 to £68,598 between mean and median survival [68], whilst the cost-effectiveness (ICER) of adding bevacizumab to this regimen was estimated at $727,202.589 per QALY [69] (which not unnaturally has led to its poor uptake as a first-line combination therapy in MPM.

Given the low costs associated with the use of vinorelbine and gemcitabine, can we improve of their potential utility in the clinic for the treatment of MPM? In this regard, the use of biomarkers will become imperative. In this regard, 

BAP1 status and sensitivity to cisplatin/pemetrexed, and gemcitabine, coupled with OS has recently been identified as a potentially key biomarker to identify patients that could benefit from this drug [70, 71], or to stratify patients away from cisplatin based therapeutic regimens [72], and to also potentially stratify them for vinorelbine sensitivity (albeit alongside the additional use of BRCA1 and MAD2L1).

Clearly, the use of biomarker based clinical trials such as MIST1 and MIST2 [49, 73], provide a suggestive methodology to achieve these, but the underlying pre-clinical data, patients selection, power and appropriate statistical approaches will be required. However, such basket-case or umbrella also comes with significant ethical challenges [74], and their results come with the real risk of over hype [75]. In real-terms, such an approach may require a large-scale Phase III clinical trial, with to truly determine efficacy.

Conclusion

The current state of affairs with respect to the latest clinical trials has led to excitement in the field and suggestions that “…we have finally turned the corner in our battle against this devastating disease…” [76]. We believe that the treatment options for MPM have not yet been achieved, in part as their superiority versus current standard-of-care is taken for granted despite to our minds a lack of solid evidence. We think that these sort of statements are not only baseless but can also generate hopes and expectations not underpinned by facts [26, 31, 76,77,78].

Moreover, concerns have recently been raised with respect to both the design and conduction of clinical cancer trials which imply that flawed study designs are a central issue of concern [79]. In addition, the over-hype of such flawed trials can result in pressure on health care systems to provide these expensive agents [79].

Therefore, after many years, unfortunately we conclude that more than ever we need additional pre-clinical screening of any new proposed cancer treatment, as well as a greater methodological rigor in trial statistics, and cost/effective ratio analysis to ensure the best efficacy to our patients and sustainability to Health Systems.