Presumed roles of AI in the clinical setting.

Our findings suggests that there was no common understanding of what medical AI is and what it can or cannot do among our study participants. Based on the interviewees’ deliberations, we identified four presumed roles along a continuum that medical AI could play in the stroke medicine: a) an administrative assistant; b) a clinical decision support system; c) the clinician’s right hand (advisory); d) fully autonomous medical AI (see Table 2). It should be noted that the boundaries between the presumed roles we identified are not perfectly distinct but understood on a continuum. HCPs tended to describe a wider range of roles, with 9 out of 14 describing at least 3 roles. Stroke survivors and family members, on the other hand, were more likely to describe one or two roles.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Presumed roles of AI in the clinical setting.

https://doi.org/10.1371/journal.pone.0279088.t002

AI as an administrative assistant. On the one hand, interviewees considered that AI-powered systems could assume the role of an administrative assistant, taking over repetitive and mundane tasks to free up clinicians’ time, leaving them more time for other, more demanding tasks. These tasks would mainly be administrative in nature and may, for instance, include patient documentation, data synthesis, or monitoring patient vitals. Participants suggested this type of AI-powered systems may be able to enhance interprofessional communication and the flow of information across different settings, including the clinical encounter and doctor-patient communication. By making information available at the point of care and freeing up time, it would allow providers to work at the top of their scope.

AI as a clinical decision support system. Other accounts of AI-powered systems underlined their role as a clinical decision support system, providing clinicians with a structure or protocol. Participants suggested that this could help clinicians to adhere to guidelines in a check-list manner. Perceived benefits would be that the system could point their attention to aspects they may have overlooked, and to facilitate faster and more accurate diagnosis by analyzing greater amounts of data in less time. One participant suggested that this could, in fact, also reassure clinicians because they would be basing their decisions on more robust data. Moreover, participants viewed an AI-based system as a possible source of inspiration on what to try out, specifically in the rehabilitation context, to provide tailored treatment to patients and to identify attainable rehabilitation goals (akin to the idea of a digital twin [34]).

AI as the clinician’s right hand. While many participants stressed that clinicians could and should interpret the system’s output (and should not blindly trust the system’s recommendations), many considered that AI could expand or complement clinicians’ expertise and capabilities in an advisory function. Such accounts suggest the role of AI as the clinician’s right hand, or in other words, a complement to or extension of the healthcare professional. This type of tool could guide clinicians when presented with particularly complex cases (e.g., orphan diseases) or in time sensitive or resource constrained settings, such as the ICU, overnight shifts, or in case of staff shortage. In this way, it could assist less experienced or well-trained clinicians in the decision-making process, or even act as a second opinion. These findings suggest it is unlikely that AI would replace expert clinicians, but rather help to compensate for lack of time or expertise in resource-constrained settings. Moreover, participants cautioned that if considered an advisor to the clinician, system recommendations might less likely be challenged by clinicians.

Fully autonomous medical AI-systems. While frequently mentioned as a potential use of AI, few participants viewed fully autonomous medical AI-systems as realistic in the near future. In this context, robotics (e.g., care robots, surgery robots or therapy robots) was the most frequently mentioned area of application in the clinical setting. During the interviews, some participants occasionally referred to “decisions made by the AI”. However, it is unclear whether these implicit attributions of decision-making power were intentional or not. One clinician, for instance, described how an AI system could not only help to exclude differential diagnosis but could also take on tasks like ordering examinations. From this description it was unclear whether, in the view of the participant, any intervention from the clinician was foreseen to approve the AI’s recommendations.

Opportunities and potential benefits of medical AI.

Most participants had a positive attitude towards medical AI and echoed the commonly promoted opportunities and potential benefits of medical AI that are reiterated in the public and academic discourse: accuracy, speed, and efficiency. Some participants explicitly linked these perceived benefits to an increase in patient safety and quality of care.

“I think this [medical AI] is good, because it means that data can be collected again for the future, which can make any programs more precise, which can set the focus more precisely for an evaluation and then a recommendation as to what is good for the individual in order to get well again [after suffering a stroke].” (Pat8)

In addition, some participants further suggested or hoped that AI may lead to more accessible, objective and consequently fair healthcare as it may compensate for human bias, healthcare providers’ preconceptions about certain individuals, or lack of well-trained staff.

"I just hope from something like that, so from computer programs or algorithms actually, that assumptions that we, I think, always make in everyday life as humans—because someone is old or somehow looks like that or is old on paper—will be less incorporated [in the decision-making process], so this subjectivity." (HCP13)

When talking about the potential gains in objectivity AI-based CDSS might offer, a clinician reflected further on fairness and how it may impact the decision-making process:

" I don’t want to accuse anyone of consciously using more diagnostics or less diagnostics in a certain way because of sympathies or something like that, or because of some conscious decision about how vital someone seems, but there is perhaps a danger, and this is where artificial intelligence could help to make things fairer, so to speak. The question is whether it is always more humane. […] if we consider justice, we have quite different definitions, which makes it more difficult again, so when does someone have a right to certain things and when rather less. So, is it fair, if someone has a genetic risk, he should get more than if someone gets the stroke maybe because of supposedly self-inflicted things? But there again the question is, with a nicotine addiction, for example, to what extent is there a free will at all? And to what extent is not also the addiction determined by some genetic aspects or environmental factors, where the patient can’t do anything about it?" (HCP5)

Will AI solve existing problems or create new ones?

Several participants cautioned that AI should not be hailed as a panacea that will solve all of healthcare’s problems, raising the question: Will AI solve existing problems or create new ones? Some suggested that AI might, in fact, not change all the much about current practices as it was just a new tool that would not impact principles such as patient autonomy, justice, or trust. Others expressed concern that medical AI might not be able to address some of the existing “most basic” challenges faced in stroke medicine and healthcare more generally, e.g., economic pressure, existing inequalities, and changing individual health behaviors:

“We’re not bad at high-tech medicine, but we can’t get a grip on normal medicine. […] The normal being the care of the population with family doctors, with specialists–it’s a disaster.” (Pat1)

“I would rather assume that the problem is not that you have the wrong options [provided by the AI system], but rather that you generally lack the resources to properly implement the options that are available. So, for example, sufficient physiotherapy in the outpatient area or something like that. That a computer-aided decision or simulation of different options would not change anything about the problem that already exists. would not change the problem that already exists.” (HCP4)

Interviews with stroke survivors and family members were dominated by personal narratives and challenges faced in the aftermath of stroke, ranging from bureaucratic hurdles (e.g. negotiations with insurers) to interpersonal challenges (e.g. activities of daily living, such as getting dressed or pursuing a professional career) and mental health problems that they did not expect medical AI would change.

A priority all three stakeholder groups voiced was the need for access to relevant patient information at the point of care to improve efficiency, treatment, and patient safety. Having to provide information repeatedly was described as a particularly frustrating experience by patients and family members.

“I think it’s completely unnecessary to have to fill out a questionnaire at every new doctor’s office, I think it’s just redundant, because the data is stored at every doctor’s office anyway, so you can actually store it right on the card and have it read out there and then.” (FM1)

Finally, some participants expressed concerns that AI may introduce new problems.

“Well, one shouldn’t overestimate AI, I have a feeling. It’s not the solution.. […] Nobody is thinking, should we really do this? Do we need to do that? And what are the long-term consequences? And that’s where I think we tend to go too far, especially in healthcare, and by [introducing] potential solutions or improvements often we create new problems, which you can’t really anticipate.” (HCP0)

AI or not, data is key.

Most participants considered comprehensive patient data pivotal to improving healthcare services in stroke medicine, whether an AI system was involved in the collection or analysis of such data seemed to be secondary. While some considered that AI-powered systems could help to collect, synthesize, and manage patient and research data, others did not explicitly mention or did not believe that AI would be needed for these types of tasks.

“Digitization leads to a more barrier-free flow of data, but that’s not necessarily artificial intelligence, but it is about the precondition that information is used in such a way that it benefits the individual patient or the doctor or the nurse because it provides more information or processes it in such a way […].” (HCP1)

Only one of the participants cautioned against what he negatively referred to as data faith, arguing that generating more data might also lead to insecurities and may in fact not be the solution to improving healthcare.

“The more data we have, the greater the uncertainty. And I think the big problem we have now is simply this, too much faith in data, or thinking that if we just collect enough data, we can improve everything. I think we should just shift down a gear there." (Pat2)

Importance of relational aspects in healthcare.

Relational aspects in healthcare featured heavily in our interviews, especially in the context of rehabilitation. Many participants considered that the importance of trust in clinical care would ultimately shape the impact of AI on the doctor-patient relationship and medical decision-making. More specifically, most were convinced that AI would not be able to replace humans in healthcare, because healthcare relies on human relationships, empathy, and human warmth. Stroke survivors in particularly highlighted the healing, therapeutic effect of having someone to talk to and feeling understood.

“There was a phase when I wasn’t doing well psychologically, and I was just happy to be able to talk to someone. And that is also good for the language, to talk about it in depth. And that was very important. To withdraw and if you only do that with the AI, I don’t see that.” (Pat12)

Stroke survivors and clinicians voiced concerns that the adoption of AI in clinical practice might slowly lead to dehumanizing healthcare, steering healthcare professionals to lose sight of the person in front of them and having patients worry about becoming just a number in the system.

“I can imagine that there is a danger that health professionals will rely more on artificial intelligence and perhaps fixate on it and pay less attention to the patients and their wishes.” (HCP8)

Some participants were also convinced that efficiency gains owed to AI would not directly benefit patients but instead lead to further streamlining i.e., a reduction in healthcare personnel. Sharing his frustration, one of the stroke survivors criticized trends of outsourcing interpersonal tasks that are central to healthcare to volunteers, painting a grim picture of the future of healthcare:

“[When volunteering], I see again and again how much this human contact is needed and desired, and I also see the efforts in healthcare to automate more and more to save personnel costs, and to shift this task to volunteers. […] Here everything is being streamlined and I can’t imagine more time being devoted to the patient because some of the work is being shifted to artificial intelligence. […] If we think about the developments over twenty or two hundred years into the future, it’s actually going to be Starship Enterprise, that the patient lies in a sterile room and if he’s lucky he gets to see a human being once a day or once a week.” (Pat2)

Roles and responsibilities.

Losely related was the subtheme roles, and responsibilities which captured participants views on some of the central topics regarding the practical implementation of AI, including questions of decision autonomy, transparency, responsibility, and fairness. Underscoring the clinician’s epistemic authority, most of the participants agreed that decision-making power should rest with clinicians and patients, and that humans should always be able to intervene, whether to confirm or to correct the AI system’s output.

“I think that as a therapist you should still have the possibility to intervene to be able to correct certain errors. And that you can take this, which has now been taken out of thousands, that you can still individually adapt it to the patient, that you can take it as evidence, but then also incorporate personal experience and intuition, to create and implement the special rehabilitation program for the patient.” (HCP9)

Going a step further, some participants cautioned that AI-powered systems may lead clinicians to second-guess themselves, doubt their own intuition, or even blindly trust the system thus neglecting their medical responsibility. Some also considered that overreliance may lead to a loss of expertise and competences in future generations of clinicians and, consequently, dependence.

"Yes, definitely. So, I think you also have to look, of course, that the treatment team then also does not rely one hundred percent on artificial intelligence, but always questions that again or also listens to the family." (FM1)

“[A risk is] that it affects the training of physicians in such a way that in the end everybody relies on it, and nobody dares to stand up and have their own opinion. That is now also generally the problem, which sometimes strikes me. When we are actually completely sure what the clinical diagnosis is, we still do an MRI in so many cases where it is not necessary, because we somehow don’t trust ourselves […]. I can imagine that this would also be a risk that we sell ourselves short, or at some point also have to, because we really can no longer do it.” (HCP6)

Related to considerations of decision-making power, medical error and responsibility were discussed. While some participants were hopeful that AI systems could reduce or prevent medical error, others recognized that both humans and AI can err, highlighted potential sources of error, like biased data sets or errors during data entry.

"And either way, mistakes can happen, so the computer system can also somehow give a recommendation that is ultimately wrong, because some parameters were entered incorrectly, because it makes a mistake." (HCP4)

There was wide agreement that the responsibility for medical decisions alongside the decision-making authority presently rests with the healthcare professional or care team. While some participants advocated that at least a rudimentary understanding of the AI system would be necessary to ensure quality and patient safety, others described the lack of explainability as not unique to medical AI, providing examples of “black box” tools currently in use.

“One must not relinquish one’s responsibility. At the same time, this is already happening, for example, with this CT monitoring. I would never check what this program has presented to me. But I don’t know of any case where there was a problem or the patient had an arrhythmia after all, and then it turned out afterwards that the program didn’t identify it. But in this respect, I actually defer my responsibility to the reading of the monitoring. Not responsibility, because if there is an error, then I still have to take the blame. But I have no way of verifying that [the output]. Responsibility is definitely on the doctor.” (HCP4)

Yet, some participants also acknowledged that forms of shared or developer responsibility might be possible in case of a malfunctioning system, for example.

"In my eyes [the doctor bears] the responsibility, yes, who else should it be? Well, we can’t hold an IT specialist responsible. We can only hold him responsible if he programs a system error or something." (Pat7)

Patients’ rights to information and decision-making about medical AI.

The third subtheme we identified captured participants’ views concerning patients’ rights to information and decision-making, reflecting considerations related to patient autonomy, data protection, and privacy. Many participants explicitly stated that patients (or their designated proxies) should have a say when it comes to data collection and medical decision-making, one proposing organ donation as an analogy.

"I think, there must be very clear information, what the programs can do and where the limits are and now comes again so the question, I do not know now, what is already discussed there, I think that must be quite transparent. And then it must be a clear consent, so as I say as a patient or as a person, okay, I agree to my organs being donated or so, it must also apply to such a situation that I agree to be tracked and I must also be able to revoke that." (FM6)

In contrast, only few pointed to potential limits of patient autonomy when it comes to the choice of tools and resources used by clinicians.

"You don’t ask the patient whether the doctor is allowed to use computers or what literature he reads or is allowed to read. I can’t imagine that, so I think the doctor must be free to decide which procedures he uses, and the patient shouldn’t be allowed to have a say in that either." (Pat2)

There was some disagreement on whether patients should be informed if an AI system was used to guide their care. Some argued that informing patients was a (moral) obligation, like informing about medication side effects, and could promote trust.

"Probably it does make sense to inform the patients that the prediction that has been made, how that has been generated so roughly. So, what patient data from how many patients went in there and normally that would be evaluated yes probably how good the evaluation is, what it has shown, how reliably the program can predict what, what errors are possible there. It probably makes sense [to inform the patient] like how you have to inform about side effects of a drug, to give a brief information overview, if you give a clear prognosis, so to speak, on what it is based on." (HCP3)

Others considered disclosure unnecessary or simply challenging, concluding that too much information might in fact be harmful, confuse and overwhelm patients, or even erode their trust, especially in the acute setting. Reiterating the importance of trust in the doctor-patient relationship, one participant emphasized that information should not be overrated.

"Well, I am of the opinion that today this demand for absolute information is a bit stupid. So, I must say, in such a case or otherwise also in normal life, the word trust is something that is very, very important." (Pat7)

Several participants voiced concerns regarding data sharing, also independently of AI, including the risk of surveillance, loss of privacy, and information getting into the wrong hands, like hackers, banks, employers, or insurance companies. However, there were also a few participants who either did not attribute much value or concern to data protection and privacy or simply considered it an illusion. Participants often tended to weigh the risks of data sharing and subsequent use of data for medical AI against potential personal or public health benefits.

"FM5: There are people in our circle of acquaintances for whom data protection is very important. We’re not quite one of them, for us it’s simply more important that the data is processed in a way so that it can be used for something. We are both no longer in professional life. That means that this could not be exploited by the employer, who then says, I now know about you, you have this and that, we’d better separate, so in this respect, I don’t know why I would have to be afraid. What do you say, [name of partner]?

Pat13: Yes, I feel the same way. Above all, we want to serve the success of the cause."