Background

Current scenario

Until recently, the approach to develop a CAD system to extract meaningful features and infer a diagnosis was based heavily on Rule Based algorithms [

Methods

Dataset creation

A pool of over 2000 postero-anterior view chest radiographs from the out-patient and in-patient department acquired on different computed radiography and digital radiography systems were assessed for quality, level of penetration, positioning, and contrast. Those with very poor quality, low contrast, and unsatisfactory positioning were rejected. However, chest radiographs with minor imperfections in breath-holding, positioning or contrast, deemed reportable by the radiologist were included. Chest radiographs with clothing, jewelry, and implantable medical devices artifacts were included to mirror real-world variations.

Image processing

The resultant dataset of 637 images were then converted from proprietary file types into Joint Photographic Expert Group file type with a 1024 × 1024 matrix size with 96 dpi vertical and horizontal resolution encoded using baseline DCT Huffman coding. The bit depth (bits per sampling) was set at 8bits with a Chroma subsampling Y’CbCr=4:2:0. The dataset was de-identified and was compliant with the Health Insurance Portability and Accountability Act. No data augmentation procedures were performed.

Model implementation

The dataset was uploaded onto Google Cloud Platform (Google LLC, Menlo Park, CA, USA) and processed using Cloud AutoML Vision Beta (release date: July 24, 2018). Multiple labels were created for classifying different pathologies and image characteristics (Fig. 1). Each image was labeled with one or more labels using Vision UI running on Chrome v68.0.3440 (Google LLC, Menlo Park, CA, USA). The dataset was randomly subdivided with 80% of the images under each category allocated to the training set and 10% each to the validation and testing sets. The system used had Intel Core i3 – 4005U 1.70 GHz chipset (Intel, Santa Clara, CA, USA), 4.00 Gb RAM, 512 Gb hard disk space, integrated Intel HD Graphics 4400 (Intel, Santa Clara, CA, USA) with Windows 7 Ultimate Operating System (Microsoft Corporation, Redmond, WA, USA).

Fig. 1
figure 1

Flowchart summarizing the data categorization and model development process

Full size image

Statistical analysis was performed with the in-built metrics projection in Vision API, Google Sheets (Google LLC, Menlo Park, CA, USA) and MedCalc (MedCalc Software Ltd, Ostend, Belgium).

Results

Dataset characteristics

The dataset contained 637 postero-anterior view chest radiographs of which 332 had some pathology (52.1%). The dataset had a mild male predominance (57.8%) with an average age of 26.5 years. Each image assessed subjectively for quality and marked either satisfactory or poor. 82.1% of the images were of satisfactory quality but the dataset also contained 17.9% radiographs which were poor in quality but still deemed reportable by the radiologist. 47.6% of the dataset contained some form of artifact from clothing, jewelry, or implantable devices like pacemakers. The images were also assessed for positioning of the subject and revealed 25.9% to have some degree of rotation—which could lead to certain artifactual findings such as apparent cardiomegaly and prominence of the hila. Forty-three of the 637 radiographs were found to have been acquired in mid-inspiration. These imperfect images were introduced into the dataset to reduce overfitting of the model to the training set and improve its real world applicability.

The images with pathology were sub-classified and labeled into 9 different categories (Fig. 2). The pathologies were also assessed for subjective conspicuity. Each lung field was divided into three lung zone: upper, middle, and lower. A pathology occupying more than or equal to half of a zone was deemed “Apparent.” If the pathology occupied less than half but more than 25% of the lung zone, it was marked as “Conspicuous.” Lesions occupying less than 25% of a lung zone were termed “Subtle.” The distributions of the lesions are shown in Fig. 2.

Fig. 2
figure 2

Distribution of conspicuousness of pathology in the training data set

Full size image

Accuracy metrics

The precision (positive predictive value) for all labels was 65.7% with a recall (sensitivity) of 40.1%. The auPRC (area under precision-recall curve or average precision) of the model was 0.616 (Fig. 3). The precision and recall for each category is summarized in Table 1. The F1 Score for classification was 0.65 for “Normal” category and 0.75 for “Pathology” category. Further evaluation statistics for both categories are summarized in Tables 2 and 3, respectively.

Fig. 3
figure 3

Precision-recall tradeoff graphs for classification of “Normal” (a) and “Pathology” (b)

Full size image
Table 1 Accuracy metrics
Full size table
Table 2 Evaluation of model performance in detecting “Normal” chest radiographs
Full size table
Table 3 Evaluation of model performance in detecting “Pathology” chest radiographs
Full size table

Discussion

Unmet needs

While there has been considerable interest in the application of convolutional neural networks and other forms of machine learning for classification of chest radiographs into various pathologies, the underlying technology utilized in all these studies remain exclusionary [5, 24,25,26]. These studies either constructed and trained machine learning models de novo or worked with pre-trained CNNs like AlexNet and GoogLeNet [3, 27]. Though these methods yielded high accuracy models which could classify chest pathologies, they were built on systems which required high level of expertise as well as prohibitively costly infrastructure. This has led to a data-algorithm divide. The predictive accuracy of an algorithm is strictly contingent on the dataset that it is trained on (Fig. 4). But a large number of institutions in resource-limited settings may not have access to machine learning technology but do have access to large volumes of data.

Fig. 4
figure 4

Examples from the training data set. Right-sided pleural effusion (a). Fibrosis in left upper zone (b). Consolidation in bilateral lung fields (c). Multiple pathologies in single radiograph—showing patchy consolidation with fibrotic bands and bulla in left upper zone (d). Cardiomegaly (e). Left hilar prominence (f). Prominent bronchovascular markings (g). Emphysema (h). Blunted left costophrenic angle with patchy consolidation (i). Collapse of left upper and middle zone (j). Nodular opacities in right lower zone (k). Normal chest radiograph with clothing artifact (l)

Full size image

Proposed solution

In this study, we tried to explore the possibility of repurposing general purpose automated machine learning models to classify diagnostic images, in particular chest radiographs. The platform used was Cloud AutoML Vision, which circumvents the challenges of requiring a large amount of time and expertise in crafting a neural network by using reinforcement learning [23]. The “controller” recurrent network creates variable length strings. These strings act as templates for development of “child” convolutional neural networks. These “child” networks are trained on the dataset and subsequently evaluated for accuracy. The accuracy metric is used as a positive reinforcement for the “controller” network. Thus in the subsequent iteration, the “child” networks with higher accuracy are favored. This is repeated until a single best “child” network is achieved with the highest accuracy.

Model accuracy

The accuracy metric of our trained model was expectedly lower than dedicated CNNs. The model had very poor sensitivity for sub-classification of pathology. However, the overall accuracy achieved for detection of pathology in chest radiographs was 74.57%. The accuracy parameters of the model are compared with two studies conducted with comparable machine learning models in Table 4.

Table 4 Comparison of accuracy metrics
Full size table

Our model, DeepDx, was able to achieve comparable accuracy to the model used by Bar et al., even surpassing their precision rate by almost 25%. This is substantial progress, especially when viewed in the context of the highly specialized fusion model (two separate deep learning baseline descriptors used along with GIST descriptor) created by Bar et al. [10]. The comparison table also reveals that Cicero et al. in their study achieved a much higher overall accuracy, but the success could be attributed at least in part to the large dataset on which their model was trained [11].

Justification

In our model, the three categories with examples above the minimum recommended number did provide good accuracy and with targeted increase in the dataset in subsequent iterations the overall model accuracy is likely to improve further. As per documentation released with Cloud AutoML Vision (Google LLC, Menlo Park, CA, USA), which we utilized in the study, the minimum recommended examples per label is 100 and approximately 1000 examples are advised for accurate prediction. This may not always be feasible for medical imaging, as rarity is often a feature of diseases with serious implications; and the time required to accrue enough examples may impede progress. This problem is usually circumvented by data augmentation procedures. Application of techniques such as horizontal flip**, crop**, rotation, and padding on chest radiographs and their effect on model accuracy has not been investigated. It may not be prudent to shoehorn techniques, while efficacious in other image datasets, onto diagnostic images. For example, horizontal flip** of a chest radiograph may create false-positive results for detection of cardiomegaly and may in fact reduce accuracy. The model may also fail to flag cases of dextrocardia. Similarly, training machine learning models on rotated radiographs may lead to the algorithms assigning undue importance to irrelevant components of the image. Also many disease processes are defined by their orientation like cephalization of vessels in CCF, which may be lost during the augmentation process.

Accuracy of the model is also likely to gain from changing the labeling structure of the dataset. In our study, we trained the algorithm to diagnose “Normal” and “Pathology” not as a binary alternative but as distinct classification categories. This was done kee** in mind real world application, as many radiographs do not distinctly fit either into an apparently normal or disease category. Many radiographs have suspicious features which should not be classified as disease and may require consensus reporting by radiologists. Another advantage of detecting the two categories separately was that it gave us comparable statistics with a larger number of studies, as most have trained to classify either one of the categories. The downside of this labeling structure was that it added to the complexity and thus probably reduced accuracy of the model. The model can be trained, in further studies; to detect only “Pathology” and the “Normal” can be processed as a default class. The sensitivity of the “Pathology” label should be increased to commit false positives and catch the indeterminate cases rather than being labeled “Normal” (Fig. 5). This will again entail human intervention to sort through and weed out the false positive, but will improve accuracy.

Fig. 5
figure 5

a False positive—the algorithm misinterpreted the artifact as nodular opacity. b False negative—the algorithm failed to identify the pleural based mass as “Pathology”

Full size image

Reflections

The study has highlighted certain definite advantages of using automated machine learning in develo** diagnostic classification models. The method reduces infrastructure requirements and cost to a fractional amount. The ease of use, with GUIs, also enables implementation and fine tuning without cumbersome coding languages. The reinforcement-based learning model greatly reduces the time requirement for develo** complex CNN architecture. And importantly, such platforms provide scalability to improve upon a model and add further complexity to the classifier.

Future implications

Further work needs to be done with larger datasets of diagnostic images, to ascertain the maximal overall accuracy achievable. Multiple platforms now exist providing similar tools and they should be evaluated in a controlled trial for unbiased comparison. Data augmentation procedures should also be validated for use with medical imaging, particularly radiological images. Lastly, most studies attempting to classify chest radiographs have dealt with post-processed compressed images converted to non-native file types such as JPEG and PNG [11, 12, 17, 18]. This conversion may lead to loss of important image characteristics and attempts should be made to use DICOM file types for future training of algorithms.

Conclusion

Computer vision is revolutionizing the field of diagnostic imaging. But its resource-intensive nature may preclude its wider implementation and acceptance. This study presented an alternative to the traditional machine learning infrastructure and aimed to investigate the use of commercially available general purpose cloud-based automated machine learning for detection of pathologies on standard postero-anterior chest radiographs.

The study found automated machine learning to be a viable alternative to human designed diagnostic convolutional neural networks. The accuracy of the model developed was conservative in comparison to standard deep learning models. However, restructuring of the classifiers and increasing the training dataset hold promise of achieving greater accuracy. Further multi-platform studies are required with larger datasets to fully explore its potential.

While machine learning promises of vast improvements in speed and accuracy in detection of pathologies across imaging modalities, greater research focus needs to be directed towards ensuring that this novel technology is used to bridge the health-wealth gap and not widen it.