Distance Based Image Classification: A solution to generative classification’s conundrum?

Abstract

Most classifiers rely on discriminative boundaries that separate instances of each class from everything else. We argue that discriminative boundaries are counter-intuitive as they define semantics by what-they-are-not; and should be replaced by generative classifiers which define semantics by what-they-are. Unfortunately, generative classifiers are significantly less accurate. This may be caused by the tendency of generative models to focus on easy to model semantic generative factors and ignore non-semantic factors that are important but difficult to model. We propose a new generative model in which semantic factors are accommodated by shell theory’s Wen-Yan et al. (IEEE Trans Pattern Anal Mach Intell, 2021) hierarchical generative process and non-semantic factors by an instance specific noise term. We use the model to develop a classification scheme which suppresses the impact of noise while preserving semantic cues. The result is a surprisingly accurate generative classifier, that takes the form of a modified nearest-neighbor algorithm; we term it distance classification. Unlike discriminative classifiers, a distance classifier: defines semantics by what-they-are; is amenable to incremental updates; and scales well with the number of classes.

Appendix

The main body focuses on develo** the best possible distance classifier. In the appendix, we relate our proposed solution to some popular nearest-neighbor alternatives.

1.1 Nearest-Neighbor Classification

Finally, no discussion of distance based classification can be complete without mentioning the nearest-neighbor classifier. This section introduces reference based noise cancelled nearest-neighbor classifier. Figure 8 shows that (similar to our other distance based classifier), noise cancellation greatly improves the nearest neighbor classifier’s validation capability. The derivation is provided below.

Fig. 8

Nearest-neighbor classification on STL-10 (Coates et al., 2011) with raw and reference based noise cancelled distances. Noise cancelled distances provide significantly better validation, as shown by the AUROC score

Table 7 Comparing distance classification with traditional nearest-neighbors

Let ${\mathbf {E}}$ be the ideal, common generator of all instances in a dataset and ${\mathbf {m}}$ be its mean. As ${\mathbf {m}}$ is constant relative to the process of generating individual instances, from Eq. (3) the distance of all ideal instances to ${\mathbf {m}}$ will be a constant, which we denote as $c_{\mathbf {m}}$. i.e. , if ${\mathbf {x}}_t, \, {\mathbf {x}}_{t'}$ are two ideal instances

$$\begin{aligned} a.s. \quad \Vert {\mathbf {x}}_t - {\mathbf {m}}\Vert = \Vert {\mathbf {x}}_{t'} - {\mathbf {m}}\Vert = c_{\mathbf {m}}. \end{aligned}$$

(44)

From Eq. (16), the distance of the noisy features ${\mathbf {x}}(t), \, {\mathbf {x}}(t')$ from ${\mathbf {m}}$ is:

$$\begin{aligned} a.s. \quad&\Vert {\mathbf {x}}(t) - {\mathbf {m}}\Vert ^2 \nonumber \\&\quad \approx \Vert {\mathbf {x}}_t - {\mathbf {m}}\Vert ^2+ \Vert {\mathbf {n}}(t)\Vert ^2 = c_{\mathbf {m}}+\Vert {\mathbf {n}}(t)\Vert ^2 , \nonumber \\ a.s. \quad&\Vert {\mathbf {x}}(t') - {\mathbf {m}}\Vert ^2 \nonumber \\&\quad \approx \Vert {\mathbf {x}}_{t'} - {\mathbf {m}}\Vert ^2 + \Vert {\mathbf {n}}(t')\Vert ^2 = c_{\mathbf {m}}+\Vert {\mathbf {n}}(t')\Vert ^2. \end{aligned}$$

(45)

From Eq. (17), the distance of ${\mathbf {x}}(t)$ from ${\mathbf {x}}(t')$ is

$$\begin{aligned} \Vert {\mathbf {x}}(t) - {\mathbf {x}}(t')\Vert ^2 \approx \Vert {\mathbf {x}}_t - {\mathbf {x}}_{t'} \Vert ^2 + \Vert {\mathbf {n}}(t)\Vert ^2 + \Vert {\mathbf {n}}(t')\Vert ^2. \end{aligned}$$

(46)

Thus, combining Eqs. (45) and (46), a noise cancelled nearest-neighbor distance be can be defined as:

$$\begin{aligned}&f^2({\mathbf {x}}(t), {\mathbf {x}}(t'), {\mathbf {m}}) \nonumber \\&\quad = \Vert {\mathbf {x}}(t) - {\mathbf {x}}(t') \Vert ^2 - \Vert {\mathbf {x}}(t) - {\mathbf {m}}\Vert ^2 - \Vert {\mathbf {x}}(t') - {\mathbf {m}}\Vert ^2 \nonumber \\&\quad \approx \Vert {\mathbf {x}}_t - {\mathbf {x}}_{t'}\Vert ^2 - 2c_{\mathbf {m}}, \end{aligned}$$

(47)

where $f^2({\mathbf {x}}(t), {\mathbf {x}}(t'), {\mathbf {m}})$ approximates the ideal squared distance with a constant offset, $-2c_{\mathbf {m}}$.

In terms of practical effectiveness, reference based noise canceled nearest-neighbor, is similar to that of standard normalization. However, the ability to achieve a normalization like effect with a different algorithm, helps validate our theory. It also provides a chance to reconsider, the classic algorithm of undergraduate textbooks, from a new perspective.

1.2 Distance vs Nearest-Neighbor Classification

Finally, it would be instructive to compare our distance classifier with a traditional nearest neighbor classification algorithm. For this task, we consider both euclidean and cosine distance classifiers with and without normalization/centering. They against our distance classifier in Table 7, which reports both classification accuracy and validation AUPRC.

As predicted, all the nearest-neighbor classification accuracies are respectable, with only minor variations across algorithms. However, there are large differences in AUPRC. For Euclidean nearest-neighbor, normalization significantly improves AUPRC, which shell theory and Sect. 4.3 explain in terms of a noise cancellation effect. A similar improvement occurs when applying the cosine distance to centered data-points. However, we do not offer an analytical explanation because cosine distances are not translational invariant; and thus, cannot be trivially analyzed using shell theory.

Although normalization significantly improves validation AUPRC of traditional nearest-neighbor algorithms, the scores remain significantly below that of our distance classifier. If normalization cannot be employed (such as in the context of strictly incremental learning), our distance classifier will have significantly high validation AUPRC than either of the traditional nearest-neighbor techniques.