Danger from chatbots: Do they need regulation as a medical device?

Chatbots like ChatGPT have great potential for medicine. But their unregulated use in healthcare poses a great danger because of their inherent risks. A Nature Medicine article by Prof. Stephen Gilbert et al. calls for medical device approval for chatbots and applications based on them. In an interview with MEDICA-trade-fair.com, Prof. Gilbert describes possible dangers.

Prof. Stephen Gilbert: Search engines with chatbots based on Large Language Models (LLMs) are increasingly used by medical professionals. However, these chatbots do not work like traditional search engines. The chatbots hide behind laws that allow search engines to list search results without being responsible for the list of websites delivered to users. These chatbot tools offer support through auto-completion. However, this no longer allows users to work completely independently. The AI first makes small suggestions and thus comes - on the doctor's shoulder, so to speak - into the consulting room. The overlap between pure communication functions and medical product functions is very small.

With the introduction of ChatGPT, an arms race has begun among the big technology companies, who want to use it to make rapid progress in medicine as well but ignore the regulations. There are already LLM-based tools that have been developed and approved specifically for medical purposes. They provide accurate and reliable information. Search engine chatbots also tend to hallucinate answers, they change the answer and the accuracy of their answers decreases. This increases the risk of harmful advice.

Are there areas where the use of chatbots would be useful?

Gilbert: Screening for possible rare diseases is a challenging task based on textual information. There are records that go back many years. This is certainly where the big language models are put to better use than, for example, in emergency situations.

In the US, there is FDA guidance that says stricter regulation is needed where physicians have little time to respond. A more open regulation is possible when the practitioner has a lot of time to think about the answers.

What is the best way to determine potential uses of LLM chatbots?

Gilbert: This technology offers phenomenal potential in almost all medical fields but also enormous risk. In its initial, uncontrolled state, it is useless. Like taming a wild mustang, you want to achieve adaptation without blocking all the potential.

There needs to be a public debate about whether these technologies are acceptably safe. This should not be an "anything goes" debate, nor should it be a "too big to regulate" debate. The result should be fair rules that apply to the whole sector. And it should not be the case that certain companies are simply too big to regulate.

Are there differences in the regulations in different countries?

Gilbert: Almost all countries are members of the World Health Organisation and follow its guidelines. Therefore, certain criteria must apply to medical devices. For large medical language models, manufacturing companies have to prove that they have control over what is used as a source.