Synthesizing Hard Training Data from Latent Hierarchical Representations

Abstract

This paper introduces a framework for creating augmented hard samples, which are new images created to resemble those that a classifier will struggle to classify. This is used for data from an automatic visual defect inspection system, specifically images of vials with and without chipped glass. The hard samples were found by training ConvNeXt classifiers and using the confidences of the classifiers on the training dataset. VQ-VAE2 was used to obtain the latent representations of the hard samples, and a PixelSnail model was used to create new high-frequency details while retaining low-frequency details. The PixelSnail model was pre-trained on a large amount of non-defect images. The augmentation method was applied to a dataset of 200 images and was evaluated by training classifiers to test the effect of using the augmented hard samples. Introducing the augmented hard samples into the dataset was found to improve classifier performance, measured in Area Under Curve (AUC) of the Receiver Operator Characteristic (ROC) curve, from 0.953 to 0.973. The method was tested with augmenting both defect and non-defect images, and the somewhat surprising conclusion is that while using augmented defect images didn’t yield improvements, augmenting non-defect images did.