One of many challenges in the field of AI is determining what constitutes evidence of impact and

benefit for AI medical devices and who should assess the evidence [²]. The majority of AI studies are conducted in experimental conditions and based on preselected data. They might provide inadequate insight into the use of AI applications in heterogeneous, real-world care settings.

Lee et al. tested seven algorithms being used clinically around the world, including one with

US Food and Drug Administration (FDA) approval and four whose developers have submitted applications for FDA approval. They found that most of these algorithms performed worse in real-world, compared with experimental, situations, with only three of seven and one of seven having comparable sensitivity and specificity to the human graders, respectively. Only one algorithm performed as well as human graders [¹¹]. Another of the algorithms tested performed significantly worse than human graders at all levels of DR severity—it missed 25.58% of cases of advanced retinopathy, which could have serious consequences. One of the potential hazards of the clinical use of algorithms identified in this study was the risk of applying an algorithm trained on a particular demographic group to a population that differs in factors such as ethnicity, age, and sex. Moreover, many

studies of algorithms developed with

AI have excluded low-quality images, treating them as ungradable images, and patients with comorbid eye diseases, making them less reflective of real-world conditions.

The study by Lee et al. shows the importance and limitations of the registration process of AI-based medical devices. FDA

registration

is based on a centralized system, which does not have a specific, easily accessible regulatory pathway for AI-based medical devices. FDA clears the medical devices through three pathways: the premarket approval pathway, the de-novo premarket review, and the 510(k) pathway [¹², ¹³]. The leading AI disciplines in medicine are radiology, cardiology, internal medicine/endocrinology, neurology, ophthalmology, emergency medicine, and oncology. FDA approvals of AI-based medical devices have increased steadily in recent years; there were 9 in 2015, 13 in 2016, 32 in 2017, 67 in 2018, and 77 in 2019, with the majority of devices designed for use in radiology, cardiology, and neurology [¹²]. Interestingly, 85% of FDA-approved medical devices in the years 2015–2019 were intended for use by health-care professionals, and only 15% for use by patients. The best-known, FDA-approved, AI-based medical devices in the field of ophthalmology are IDx-DR (2018), the first software to provide screening decisions that do not have to be interpreted by a clinician, and Eyenuk (2020), which, like

, screens for diabetic retinopathy.

In European Economic Area, which includes the European Union (EU) countries and the European Free Trade Association (EFTA)

members (Iceland, Lichtenstein, Norway, and Switzerland), medical devices are approved in a decentralized manner.

Conformité Européenne (CE) marking indicates conformity with EU health,

safety, and environmental-protection standards. For the lowest-risk medical devices (CE class I), the manufacturer ensures that the product complies with regulations and an approval procedure is not required. The registration procedure for higher-risk medical devices (CE class IIa, IIb, and III) is handled by private entities, called notified bodies, that have been accredited to assess the devices and issue a CE mark.

Thirteen

CE-marked AI-based medical devices were approved in 2015, 27 in 2016, 26 in 2017, 55 in 2018, and 100 in 2019. The majority were designed for use in radiology, general hospital care, cardiology, neurology, ophthalmology (12 devices), and pathology, and most were class IIa (40%), class I (35%), or class IIb (12%) devices [¹²]. Of the AI-based devices that were CE-marked between 2015 and 2019, 124 (52%) were also FDA approved, making up 56% of the AI-based tools that the FDA approved in those. Bigger companies were more likely to get both approvals, whereas smaller companies were more likely to obtain only a CE mark. The authors of this study suggested that the European approval

system was less rigorous than the US one. This conclusion is supported by an FDA report on 12 devices that received CE approval only and later were found to be unsafe or ineffective [¹³, ¹⁴]. A major problem in studying CE-marked devices in the European Economic Area is the lack of a publicly available register of approved devices comparable to the FDA register. Moreover, the information submitted to the notified bodies is confidential. In 2022, a new European database on medical devices (Eudamed), providing a live picture of the lifecycle of medical devices, will become operational. It will be composed of six modules, including actor registration, unique device identification (UDI), device registration, notified bodies and certificates, clinical investigations and performance studies, and

vigilance and market surveillance [¹⁵].

DL algorithm training requires large data sets with thousands or even hundreds of thousands of diverse, well-balanced, and accurately labeled images [¹⁶]. The

resources required for an AI study are presented in Fig. 1.6. The enormous numbers of required images rarely can be obtained from individual centers; thus they are secured from data repositories or centers that agree to share data. There is a growing need for consensus on standardized definitions of medical entities; conventions for data formatting; identification of units of measure; protocols for data cleaning, harmonization, and validation; standards for sharing and reusing data and sharing of code implementing AI models; and the adoption of open application program interfaces to AI models [¹⁷]. This is required for data sharing and open communication in AI, which is critical for conducting the reproducible research that is necessary before AI

technology can be adopted in health care.

The schematic presentation of

resources required for an AI study

Kermany et al. used a DL analysis of a data set of

optical coherence tomography images for triage and diagnosis of choroidal neovascularization, diabetic macular edema, and drusen. They demonstrated performance comparable to that of human experts and provided a more transparent and interpretable diagnosis by highlighting the regions recognized by the neural network. Further, they showed that a transfer-learning approach produced only modestly worse results (twofold increase of error, compared with the full data set) while using approximately 20 times fewer images. They also demonstrated the wider utility of this approach by applying it to the identification of pediatric pneumonia using chest X-ray images. They provided their data and code in a publicly available database to facilitate their use by other biomedical researchers in order to improve the

performance of future models [¹⁸].

Transfer learning (Figs. 1.7 and 1.8) has been used in recent years to build classification models for medical images because the number of images that can be used for training is relatively small compared to the number of images available to train general models [¹⁹] (Fig. 1.9). Another approach to meet the need for large, annotated training data sets might be the use of low-shot DL algorithms.

Low-shot learning (LSL), also known as few-shot learning, is a type of

machine learning (ML) problems where the training dataset contains limited information. It is well known that many real-life situations, including rare diseases (e.g., serpiginous choroidopathy or angioid streaks in pseudoxanthoma elasticum) and non-typical presentations or subtypes of common disorders, are prone to AI bias due to the paucity or imbalance of data. These deficiencies may also result in less accurate future models. When addressing this sort of bias, dividing data according to some patient features (e.g., age, sex, and race/ethnicity) may result in smaller data sets that may be insufficient for training models for these particular groups. The study by Burlina et al. showed that the performance of widely used DL methods degraded substantially when used with limited data sets, but LSL methods performed better and might be applied in retinal

diagnostics when a limited number of retina images are available for training [²⁰].

The idea of

transfer learning.