Double-weighted kNN: a simple and efficient variant with embedded feature selection

Abstract

Predictive modeling aims at providing estimates of an unknown variable, the target, from a set of known ones, the input. The k Nearest Neighbors (kNN) is one of the best-known predictive algorithms due to its simplicity and well behavior. However, this class of models has some drawbacks, such as the non-robustness to the existence of irrelevant input features or the need to transform qualitative variables into dummies, with the corresponding loss of information for ordinal ones. In this work, a kNN regression variant, easily adaptable for classification purposes, is suggested. The proposal allows dealing with all types of input variables while embedding feature selection in a simple and efficient manner, reducing the tuning phase. More precisely, making use of the weighted Gower distance, we develop a powerful tool to cope with these inconveniences. Finally, to boost the tool predictive power, a second weighting scheme is added to the neighbors. The proposed method is applied to a collection of 20 data sets, different in size, data type, and distribution of the target variable. Moreover, the results are compared with the previously proposed kNN variants, showing its supremacy, particularly when the weighting scheme is based on non-linear association measures.

Appendix

In this “Appendix” we further explore the different possibilities that exist regarding the treatment given to ordinal variables. The default strategy when working with this type of variables and the Euclidean distance consists of ignoring the natural order of the levels and generating several dummy variables that can be treated as interval ones afterward.

An alternative to this strategy, which is inspired by the treatment given in the Gower distance to this kind of variables, consists of replacing them with the corresponding position index in their natural ordering and then treat them as quantitative ones (which includes the standardization phase).

From a theoretical point of view, the fact that interval variables need to be standardized before use in the Euclidean distance, implies that the values the distances take are determined by the variable's variance:

$${d}_{{\text{euc}}}\left({z}_{i},{z}_{j}\right)=\sqrt{{\sum }_{k=1}^{p}{\left({z}_{ik}-{z}_{jk}\right)}^{2}}=\sqrt{{\sum }_{k=1}^{p}{\left(\frac{{x}_{ik}-\overline{{x }_{k}}}{{\text{sd}}\left({x}_{k}\right)}-\frac{{x}_{jk}-\overline{{x }_{k}}}{{\text{sd}}\left({x}_{k}\right)}\right)}^{2}}=\sqrt{{\sum }_{k=1}^{p}{\left(\frac{{x}_{ik}-{x}_{jk}}{{\text{sd}}\left({x}_{k}\right)}\right)}^{2}.}$$

This translates to the fact that, for instance, two ordinal variables (considered as interval) with the same number of levels, will contribute differently to the overall distance simply because of the differences in the variance (which, in turn, depends on the frequency of the categories of the original variable). This does not take place when using the Gower distance [see Eq. (2)], where the distance in the variables depends solely on the values that it takes (and not its distribution). As an illustrative example, let us assume that there are two ordinal variables with three categories, which are transformed into 1–2–3 according to its natural order. The first of them, X₁, has frequencies equal to 10–80–10% and the second, X₂, 33–33–33%. This leads to variances of 0.2 and 2.8867, respectively. If two observations take the furthest possible values in both variables, the contribution of both variables would be 20 and 1.3857, respectively. In other words, even when the difference among values is the same, variable X₁ contribution is more than 14 times higher than that of X₂.

Furthermore, in order to show that this strategy (coding ordinal variables as interval ones) does not overpower the classic one, we have conducted another experiment, comparing the performance of both strategies when applying kNN with the Euclidean distance. For this purpose, we have repeated the experiment explained in “Experimental results” section but only on the data sets that have at least one ordinal variable (see Table 2 to check the data sets and their order in the plot). The results are included in Fig. 5, which can be interpreted in the same fashion as Fig. 4B. As it can be seen, no significant differences arise between both strategies, except for the case of the third data set (where some improvement can be seen but not enough to surpass the Gower-based proposal, see Fig. 4B), which implies that regardless of the treatment applied to ordinal variables in the Euclidean context, the Gower-based proposal leads to better results.