Abstract
Purpose
Clinical benefits result from electronic patient-reported outcome (ePRO) systems that enable remote symptom monitoring. Although clinically useful, real-time alert notifications for severe or worsening symptoms can overburden nurses. Thus, we aimed to algorithmically identify likely non-urgent alerts that could be suppressed.
Methods
We evaluated alerts from the PRO-TECT trial (Alliance AFT-39) in which oncology practices implemented remote symptom monitoring. Patients completed weekly at-home ePRO symptom surveys, and nurses received real-time alert notifications for severe or worsening symptoms. During parts of the trial, patients and nurses each indicated whether alerts were urgent or could wait until the next visit. We developed an algorithm for suppressing alerts based on patient assessment of urgency and model-based predictions of nurse assessment of urgency.
Results
593 patients participated (median age = 64 years, 61% female, 80% white, 10% reported never using computers/tablets/smartphones). Patients completed 91% of expected weekly surveys. 34% of surveys generated an alert, and 59% of alerts prompted immediate nurse actions. Patients considered 10% of alerts urgent. Of the remaining cases, nurses considered alerts urgent more often when patients reported any worsening symptom compared to the prior week (33% of alerts with versus 26% without any worsening symptom, p = 0.009). The algorithm identified 38% of alerts as likely non-urgent that could be suppressed with acceptable discrimination (sensitivity = 80%, 95% CI [76%, 84%]; specificity = 52%, 95% CI [49%, 55%]).
Conclusion
An algorithm can identify remote symptom monitoring alerts likely to be considered non-urgent by nurses, and may assist in fostering nurse acceptance and implementation feasibility of ePRO systems.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
Investigators can submit a request for data from this trial by completing an Alliance Data Sharing Request Form, found at https://www.allianceforclinicaltrialsinoncology.org/main/public/standard.xhtml?path=%2FPublic%2FDatasharing, and submitting it to DataSharing@AllianceNCTN.org.
References
Cleeland, C. S., Zhao, F., Chang, V. T., et al. (2013). The symptom burden of cancer: Evidence for a core set of cancer-related and treatment-related symptoms from the Eastern Cooperative Oncology Group Symptom Outcomes and Practice Patterns study. Cancer, 119(24), 4333–4340.
Reilly, C. M., Bruner, D. W., Mitchell, S. A., et al. (2013). A literature synthesis of symptom prevalence and severity in persons receiving active cancer treatment. Supportive Care in Cancer, 21(6), 1525–1550.
Panattoni, L., Fedorenko, C., Greenwood-Hickman, M. A., et al. (2018). Characterizing potentially preventable cancer- and chronic disease-related emergency department use in the year after treatment initiation: A regional study. Journal of Oncology Practice/American Society of Clinical Oncology, 14(3), e176–e185.
Henry, D. H., Viswanathan, H. N., Elkin, E. P., et al. (2008). Symptoms and treatment burden associated with cancer treatment: Results from a cross-sectional national survey in the US. Supportive Care in Cancer, 16(7), 791–801.
Schmidt, T., Valuck, T., Perkins, B., et al. (2021). Improving patient-reported measures in oncology: A payer call to action. Journal of Managed Care and Specialty Pharmacy, 27(1), 118–126.
Centers for Medicare & Medicaid Services (CMS). Enhancing oncology model. https://innovation.cms.gov/innovation-models/enhancing-oncology-model.
Basch, E., Schrag, D., Henson, S., et al. (2022). Effect of electronic symptom monitoring on patient-reported outcomes among patients with metastatic cancer: A randomized clinical trial. JAMA, 327(24), 2413–2422.
National Cancer Institute. Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events. https://healthcaredelivery.cancer.gov/pro-ctcae/.
Dueck, A. C., Mendoza, T. R., Mitchell, S. A., et al. (2015). Validity and reliability of the US National Cancer Institute’s Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE). JAMA Oncology, 1(8), 1051–1059.
Al-Rashdan, A., Sutradhar, R., Nazeri-Rad, N., et al. (2021). Comparing the ability of physician-reported versus patient-reported performance status to predict survival in a population-based cohort of newly diagnosed cancer patients. Clinical Oncology (Royal College of Radiologists), 33(7), 476–482.
Stover, A. M., Tompkins Stricker, C., Hammelef, K., et al. (2019). Using stakeholder engagement to overcome barriers to implementing patient-reported outcomes (PROs) in cancer care delivery: Approaches from 3 prospective studies. Medical Care, 57(5), S92–S99.
Kiernan, K. (2018). Insights into using the GLIMMIX procedure to model categorical outcomes with random effects. SAS Technical Papers, 2018, 1–23.
Basch, E., Stover, A. M., Schrag, D., et al. (2020). Clinical utility and user perceptions of a digital system for electronic patient-reported symptom monitoring during routine cancer care: Findings from the PRO-TECT trial. JCO Clinical Cancer Informatics, 4, 947–957.
Mandrekar, J. N. (2010). Receiver operating characteristic curve in diagnostic test assessment. Journal of Thoracic Oncology, 5(9), 1315–1316.
Acknowledgements
The sponsor, Alliance Foundation Trials, LLC, coordinated contracting with participating study practice sites and submission of regulatory documentation. The sponsor was not involved in the design or conduct of the study; collection, analysis, or interpretation of the data; preparation of the manuscript; or decision to submit the manuscript for publication. The Alliance Publications Committee reviewed and approved this manuscript for publication and did not have the right to veto publication or to control the decision regarding to which journal the manuscript was submitted. All statements in this publication, including its findings, are solely those of the authors and do not necessarily represent the views of PCORI or its Board of Governors or Methodology Committee. Dr. Mazza presented results from this article at the International Society for Quality of Life Research Conference in Calgary, AB, Canada on October 20, 2023.
Funding
This work was funded by the Patient-Centered Outcomes Research Institute (PCORI) award IHS-1511-33392. The trial made use of technology systems provided by the Patient-Reported Outcomes Core (PRO Core; https://pro.unc.edu) at the Lineberger Comprehensive Cancer Center of the University of North Carolina, funded by NCI Cancer Center Core Support grant P30CA016086 and the University Cancer Research Fund of North Carolina. The trial was sponsored by Alliance Foundation Trials, LLC (https://acknowledgments.alliancefound.org).
Author information
Authors and Affiliations
Contributions
GLM and EB formulated the research question and designed the study. EB and ACD provided supervision for the research activity planning and execution. BG, ACD, and GLM prepared the data for analysis. GLM and BG analyzed the data. GLM led the writing of the manuscript. GLM created the tables and figures. GLM, ACD, BG, JJ, AMD, PC, VSB, GT, MJ, LJR, DS, and EB contributed to the conduct and primary analysis of the PRO-TECT trial. All authors interpreted the results as well as reviewed and approved the manuscript for publication.
Corresponding author
Ethics declarations
Conflict of interest
Dr. Basch reported receipt of personal fees from Navigating Cancer, Sivan, AstraZeneca, Resilience, and the Research Triangle Institute for serving as a scientific advisor; receipt of research funding from the National Cancer Institute (NCI); being owner of Carolina Informatics; and employment by the University of North Carolina. Dr. Schrag reported receipt of personal fees from Pfizer, nonfinancial support from Grail for serving as a site principal investigator, and grants from the American Association for Cancer Research awarded to Memorial Sloan Kettering Cancer Center. Dr. Mody reported consultancy and receipt of research funding from Sivan. All remaining authors have declared no conflicts of interest.
Consent to participate
Informed consent was obtained from all participants included in the PRO-TECT trial.
Ethics approval
The PRO-TECT trial was conducted in accordance with the Declaration of Helsinki and Good Clinical Practice guidelines. The trial’s protocol was approved by the central institutional review board and the institutional review board at each participating site.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Mazza, G.L., Dueck, A.C., Ginos, B. et al. Optimization of alert notifications in electronic patient-reported outcome (ePRO) remote symptom monitoring systems (AFT-39). Qual Life Res 33, 1985–1995 (2024). https://doi.org/10.1007/s11136-024-03675-3
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11136-024-03675-3