A Multi-source Domain Adaption Approach to Minority Disk Failure Prediction

Abstract

Frequent happening of disk failures affects the reliability of the storage system, which can cause jittering of performance or even data loss of services and thus seriously threaten the quality of service. Although a host of machine (deep) learning-based disk failure prediction approaches have been proposed to prevent system breakdown due to unexpected disk failure, they are able to achieve high performance based on the assumption that the disk model has plenty of samples (especially failure samples). However, new disk models continuously appear in data centers with the evolution of disk manufacturing technology and the expansion of storage system capacity. Limited by the deploying time, these disk models have few failure samples and are called minority disks. The minority disks are widespread in large-scale data centers and contain amounts of disks while existing approaches cannot reach satisfying performance on such disks due to the lack of failure samples. What’s worse, failure prediction models trained on other disk models cannot be directly applied to these minority disks either due to the commonly existing distribution shift among disk models. In this work, we propose DiskDA, a novel multi-source domain adaption-based solution that can fully utilize knowledge from other disk models to predict failures for minority disks having no failure samples. Our experimental results on real-world datasets show the superiority of DiskDA against previous approaches on minority disks with a few failure samples. What’s more, DiskDA also shows its good adaptivity on minority disks having no failure samples, whereas previous works are unusable.

7 Appendix

Proof. The discrepancy between the source and target domain is measured using the Wasserstein distance in DiskDA. Specifically, the $\mathop {p}$-th Wasserstein distance between two Borel probability measures $\mathbb {P}$ and $\mathbb {Q}$ is defined as:

$$\begin{aligned} W_{p}(\mathbb {P},\mathbb {Q}) = (\mathop {inf}\limits _{\mu \in \Gamma (\mathbb {P},\mathbb {Q})}\int \rho (x,y)^{p}d\mu (x,y))^{1/p} \end{aligned}$$

(9)

where the $\Gamma (\mathbb {P,Q})$ is the set of all joint distributions $\mu (x,y)$ whose marginal distribution are $\mathbb {P}$ and $\mathbb {Q}$. The $\mu (x,y)$ can be viewed as a policy for transporting a unit quantity of material from x to y and the $\rho (x,y)$ is the corresponding cost. And the Wasserstein distance between $\mathbb {P}$ and $\mathbb {Q}$ represents the minimum expected transport cost. As Wasserstein distance satisfies the triangle inequality, the following equation holds

$$\begin{aligned} W_p(\mathbb {P}_{s},\mathbb {P}_{t})\le W_{p}(\mathbb {P}_{s},\mathbb {P}_{t_{H}})+W_{p}(\mathbb {P}_{t_{H}},\mathbb {P}_{t}) \end{aligned}$$

(10)

Shen et al. [14] prove the generalization error bound of a classification function h in the target domain for unsupervised domain adaption based on Wasserstein distance as

$$\begin{aligned} \epsilon _{t}(h) \le \epsilon _{s}(h) + 2KW_{1}(\mathbb {P}_s,\mathbb {P}_t) + \lambda \end{aligned}$$

(11)

where the K means that all hypotheses h are K-Lipschitz continous, $\lambda$ is the combined error of the optimal hypothesis $h*$ which minimizes the combined error $\epsilon _{s}(h)+\epsilon _{t}(h)$, $\mathbb {P}_s$ and $\mathbb {P}_t$ are distributions of source and target domain, respectively. Let C denote $2KW_{1}(\mathbb {P}_{t_{H}},\mathbb {P}_{t})$. By substituting inequality (11) for (10), Theorem 3.1 is derived.