A Data-Efficient Deep Learning Framework for Segmentation and Classification of Histopathology Images

Abstract

The current study of cell architecture of inflammation in histopathology images commonly performed for diagnosis and research purposes excludes a lot of information available on the biopsy slide. In autoimmune diseases, major outstanding research questions remain regarding which cell types participate in inflammation at the tissue level, and how they interact with each other. While these questions can be partially answered using traditional methods, artificial intelligence approaches for segmentation and classification provide a much more efficient method to understand the architecture of inflammation in autoimmune disease, holding a great promise for novel insights. In this paper, we empirically develop deep learning approaches that uses dermatomyositis biopsies of human tissue to detect and identify inflammatory cells. Our approach improves classification performance by 26% and segmentation performance by 5%. We also propose a novel post-processing autoencoder architecture that improves segmentation performance by an additional 3%.

A Appendix

1.1 A.1 Expansion of Results

In Tables 6 and 7 we show complete results with mean and standard deviation. These are an expansion of Table 2 in Sect. 5.1 of the main paper. Tables were compressed to save space and only focus on the main results. To provide a complete picture, we added extended results in this section.

Table 6. This table shows the IoU score on the test set for UNet. We compared results without and with autoencoder for both ReLU and GELU activations for UNet Architecture. These results are averaged over five runs with different seed values. We observed that in all cases addition of APP improved performance. GELU activated APP seems out perform the ReLU activated APP in all cases except for ResNet-50.

Table 7. This table shows the IoU score on the test set for UNet++. These results are averaged over five runs with different seed values. We compare results without and with autoencoder for both ReLU and GELU activations for UNet++ Architecture. We observed that in most cases, APP improves performance except for UNet++ with Resnet-18, where APP segmentation techniques lag by around 5%. However, as a counter for ResNet-34 APP-based segmentation techniques are almost 10% better than UNet++ without APP.

1.2 A.2 Autoencoder with Efficientnet Encoder for Segmentation

In Tables 8 and 9 we compared the time taken to train and the performance of the respective trained architecture for segmentation using EfficientNet encoders. We observed that with the addition of autoencoder post-processing, training time increased by an average of 3 m 7.3 s over 50 epochs (averaged over the entire efficientnet family). This is an increase of 2.93% in training time over the eight encoders. In other words, an average increase of 0.36% increase in time per encoder over 50 epochs.

Performance wise architecture with autoencoder post-processing consistently outperformed segmentation architectures without them by 2.75%.

Table 8. In this table we report the running time averaged over 5 runs with different seeds, for efficient-net encoder family with UNet.The variation is almost negligible($<6s$).

Table 9. In this table we report the IoU averaged over 5 runs with different seeds, for efficient-net encoder family with UNet architecture.

Table 10. In this table we report the running time averaged over 5 runs with different seeds, for efficient-net encoder family with UNet++.

Similarly, we compared computational and performance for UNet++ with and without the autoencoder post-processing in Tables 10 and 11 respectively. In this case, we observed that the gain in performance with autoencoder post-processing is 5% averaged over the efficientnet family of encoders. This also corresponds to a 3 m 7 s increase in training time which is an increase of 2.6%.

Table 11. In this table we report the IoU averaged over 5 runs with different seeds, for efficient-net encoder family with UNet++ architecture.

1.3 A.3 Metrics Description

For measuring segmentation performance, we use IoU or intersection over union metric. It helps us understand how similar sample sets are.

Here the comparison is made between the output mask by segmentation pipeline against the ground truth mask.

For measuring classification performance, we use the F1 score.

Computed as F1 = $\frac{\text {2*Precision*Recall}}{\text {Precision+Recall}} = \frac{\text {2*TP}}{\text {2*TP+FP+FN}}$

1.4 A.4 Effect of Different Weights

ImageNet initialization has been the defacto norm for most transfer learning tasks. Although in some cases, as in [1] it was observed that noisy student weights performed better than ImageNet initialization. To study the effect in our case, we used advprop and noisy student initialization. ImageNet weights for initialization work for medical data not because of feature reuse but because of better weight scaling and faster convergence [18]. Noisy student training [26] extends the idea of self-training and distillation with the use of equal-or-larger student models, and noise such as dropout, stochastic depth, and data augmentation via RandAugment is added to the student during learning so that the student generalizes better than the teacher. First, an EfficientNet model is trained on labelled images and is used as a teacher to generate pseudo labels for 300M unlabeled images. We then train a larger EfficientNet as a student model on the combination of labelled and pseudo-labelled images. This helps reduce the error rate, increases robustness and improves performance over the existing state-of-the-art on ImageNet.

(ii)AdvProp training, which banks on Adversarial examples, which are commonly viewed as a threat to ConvNets. In [25] they present an opposite perspective: adversarial examples can be used to improve image recognition models. They treat adversarial examples as additional examples to prevent overfitting. It performs better when the models are bigger. This improves upon performance for various ImageNet and its’ subset benchmarks.

Since initially all these were developed for the EfficientNet family of the encoders, we used them for benchmarking. We present their results in Table 12.

Table 12. Using different initialization, we saw that the performance of different encoders of the EfficientNet family on UNet. We report the IoU over the test set in the following table. We observe that while performance gains for smaller models, ImageNet initialisation works better for larger models. Also, the fact that advprop and noisy are not readily available for all models, hence the choice of ImageNet still dominates.

Table 13. We report the F1 score for different initializations for the EfficientNet family of encoders. We reported the average of 6-fold runs on the test set with five different seed values. We observed that 0.8463 is the peak with ImageNet, 0.843 with advprop and 0.8457 with noisy student initialization.

Similarly, we conduct similar experiments for classification with different initialization. We reported these results in Table 13.

As we can see for segmentation, ImageNet initialization performed better in most cases. Similarly, in classification, it not only performed better in most cases but also provided the best overall result—these inferences, combined with the fact that advprop and noisy student requires additional computational resources. Hence we decide to stick with ImageNet initialization.

1.5 A.5 Expansion on Experimental Details

Segmentation. We used PyTorch lightning’s [7] seed everything functionality to seed all the generator values uniformly. For setting the seed value, we randomly generated a set of numbers in the range of 1 and 1000. We did not perform an extensive search of space to optimise performance with seed value as suggested in [17]. We used seed values 26, 77, 334, 517 and 994. For augmentation, we used conversion to PIL Image to apply random rotation (degrees=3), random vertical and horizontal flip, then conversion to tensor and finally channel normalisation. We could have used a resize function to reshape the 1408 by 1876 Whole Slide Images (WSI), but we instead tilled them in 480 square tile images. We then split them into a batch size of 16 before finally passing through the segmentation architecture (UNet/UNet++). We used channel attention only decoder, with ImageNet initialisation and a decoder depth of 3 (256, 128, 64).

We used cross-entropy loss with dark/light pixel normalization, Adam optimizer with LR set to 3.6e-04 and weight decay of 1e-05. We used a cosine scheduling rate with a minimum value set to 3.4e-04.

APP Segmentation. When using APP we used GELU activation by default with adam optimizer and lr set to 1e-3.

Classification. For Classification, we used the same seed values with PyTorch lightning’s [7] seed everything functionality, as described for segmentation above. For augmentation, we resized the images to 384 square images, followed by randomly applying colour jitter (0.2, 0.2, 0.2) or random perspective (distortion scale=0.2) with probability 0.3, colour jittering (0.2, 0.2, 0.2) or random affine (degrees=10) with probability 0.3, random vertical flip and random horizontal flip with probability 0.3 and finally channel normalization.

We used Stochastic weigh averaging with adam optimizer. We used a cosine learning rate starting at 1e-3 and a minimum set to 1e-6. We used focal loss with normalized class weight as our loss function. We used 6-fold validation with each fold of 20 epochs and batch size of 16. We used same parameters for both CNN and Transformers.