Against the backdrop of the growing impact AI is having in health care (Topol 2019), a discussion is develo** about its proper deployment in patient care. Contributing to this debate, Chan (2023) maintains that physicians should abstain from using black box AI, except under one of two conditions: (1) when they use medical AI as a ‘co-pilot’ and can establish the correctness of its results without AI assistance, and (2) when dealing with patients whose diagnosis and prognosis are very poor. Barring these two circumstances physicians should do without black box AI, because they should be able to explain AI decision-making to their patients, and the how and why of black box AI decisions are inscrutable (Chan 2023). The aim of this editorial is not to engage with the intricacies of Chan’s arguments. Instead, we wish to point out some related issues that deserve further reflection in the debate going forward.

Black box AI versus explainable AI

A dichotomy between black box AI and explainable AI suggests a black-and-white phenomenon, whereby an AI system and its results can either be explained or not. Yet, this all-or-nothing conception does not match the common sense understanding of the concept of explanation. In fact, the latter allows of various degrees of detail and comprehensiveness.

One might even argue that on this common sense account, even black box AI could be explained, although admittedly only to a fairly limited degree. Even though one does not know how the AI system arrives at its results in detail, one might for example convey certain features of its architecture at a generic level. Information could be given about its algorithms, the set of training data used, and certain characteristics of the model. Moreover, details might be shared about whether there is any clinical validation of the medical AI system, what is known about the sensitivity and specificity of its diagnoses (if anything), or how they compare to human diagnoses (if known from studies). Finally, it might also be explained why a more fine-grained understanding of the inner workings of the AI system is difficult to obtain.

Explainability requirements

In general, the appropriate level of detail and comprehensiveness of an explanation depends on the context. Domain experts might – in their own understanding of certain AI systems - aim for an in-depth account which is so detailed and comprehensive that it incudes (nearly) all the lower level causal and /or logical steps that lead to an AI system’s results. However, it would be unreasonable to require the same account when explaining these results to patients. After all, they are usually more interested in higher level explanations that are directly relevant to them and their condition, such as is necessary for a proper understanding of their diagnosis, prognosis, and the pros and cons of certain alternative treatment options. Likewise, it is not usual, nor is it considered required, to explain other sophisticated medical technologies - such as mRNA and MRI technologies - to patients with the same level of detail and comprehensiveness as would be suitable amongst domain experts.Footnote 1

Instead of explainability requirements appropriate for domain experts, suitable patient oriented requirements should be developed and agreed upon. Ploug and Holm (2020), for example, maintain that AI diagnoses should be contestable by patients. Departing from this premise, they formulate a set of specific informational requirements to do with data, biases, diagnostic performance as well as the medical decision itself. To focus on the latter, patients should not only know the relative human and AI contributions to the decision making process, but also who is responsible for the decision (Ploug and Holm 2020, p.3).

Trust

Whilst the ability to convey appropriate explanations of medical AI to patients is valuable and should definitely be aimed for, it is not the only thing that counts when it comes to establishing trust. What is more, a detailed and comprehensive explanation of the specific modus operandi of technologies is generally not required for people to have confidence in their use.

Long gone are the days when humans only engaged with completely understandable and transparent artifacts, e.g. spears, bows, arrows, and stone tools. In this day and age, we are surrounded by artifacts such as bridges, elevators, cars, trains and airplanes that we engage with and use on a regular basis without necessarily having any detailed and comprehensive understanding of their construction and functioning. We do not engage with them because we are thrill seekers. Rather we know that there are regulatory frameworks and quality standards in place as well as domain experts with appropriate credentials who maintain, control and direct these artifacts, thus assuring us that the risks of use are acceptable.

A similar system should be established when it comes to the deployment of AI technologies in healthcare. Suitable regulatory frameworks should be put in place (cf. Meskó and Topol 2023). Medical AI technologies should be carefully validated. Studies should be conducted to establish their effectiveness and risks. Adequate quality standards should be applied. Permissions for clinical deployment should be granted by appropriate institutions. Medical education should be adjusted, so as to deliver physicians able to use medical AI in a responsible manner. These are just a few elements of a wholly new ecosystem that should be established around medical AI to facilitate its responsible further development and deployment and thus instil a necessary measure of confidence in its deployment.