Heterogeneity Aware Distributed Machine Learning at the Wireless Edge for Health IoT Applications: An EEG Data Case Study

Abstract

In this book chapter, we design and develop a mobile edge learning (MEL) framework that enables multiple end user devices or “learners” to cooperatively train a machine learning (ML) model in a wireless edge environment. We will focus on designing and develo** the heterogeneity aware synchronous (HA-Sync) approach with time constraints and extend the framework to consider dual-time and energy constraints. The proposed MEL framework will include the commonly known federated learning (FL) as well as parallelized learning (PL). After discussing the system model and a brief convergence proof for both FL and PL, we will formulate the problem as a quadratically constrained integer linear program (QCILP), relax it to a QCLP, and propose analytical solutions based on Lagrangian analysis, Karush-Kuhn-Tucker (KKT) conditions, and partial fraction expansion. For the problem with dual-time and energy constraints, we will propose solutions based on the suggest and improve (SAI) approach. Results based on achievable local updates, validation accuracy progression, and the optimization algorithm’s execution time will be used to demonstrate the superiority of the proposed HA-Sync compared to the heterogeneity unaware (HU) approaches. As an application focus, we will demonstrate a real-time medical event prediction by showing the applicability of personalized MEL for epileptic seizure detection and predictions – using EEG data.

Appendices

Appendix 1

The global model’ loss is denoted by $F(\mathbf {w})$. Let us assume that this function has the following three properties:

• 1. $F(\mathbf {w})$ is convex.

• 2. $F(\mathbf {w})$ is $\rho$-Lipschitz: $\lVert F(\mathbf {w})-F(\bar {\mathbf {w}}) \rvert \leq \rho \lvert \mathbf {w}-\bar {\mathbf {w}} \rvert$

• 3. $F(\mathbf {w})$ is $\beta$-smooth: $\lVert \nabla F(\mathbf {w})-\nabla F(\bar {\mathbf {w}}) \rvert \leq \beta \lvert \mathbf {w}-\bar {\mathbf {w}} \rvert$

Each learner $k \in \mathcal {K}$ in HA-Sync performs a total of L updates. Then, the difference between the loss at update L and the optimal global model denoted by ${\mathbf {w}}^*$ is bounded by:

$$\displaystyle \begin{aligned} F(\mathbf{w}[L]-F({\mathbf{w}}^*) \leq \dfrac{1}{G\tau \left[A+B(1-C) \right]} \end{aligned} $$

(3.43)

The learning rate is given by $\eta$ and we can define a control parameter $\phi = 1-\frac {\eta \beta }{2}$. The local losses are bound by the parameter $\epsilon$ whereas the function $h(\tau ) = \frac {\delta }{\beta }[(\eta \beta + 1)^\tau -1]-\eta \delta \tau$. For more details on $\delta$ and $\epsilon$, the reader is referred to [12]. In general, $\eta$ is selected such that $0<\eta \beta < 1$, $\eta \phi - \frac {\rho h(\tau )}{\tau \epsilon ^2} \geq 0$, and $(\eta \beta + 1)^\tau \geq \eta \beta \tau +1$. Consider the case where MEL is not optimized and each global cycle allows each learner $k \in \mathcal {K}$ to perform the same integer number of local updates $\tau$. Then, in a fixed number of global updates G, MEL will allow for a total of $L = G\tau$ updates. Let us define the constants $A = \eta \phi + \frac {\rho \delta }{\epsilon ^2}$, $B = \frac {\rho \delta }{\beta \epsilon ^2}$, and $C = \eta \beta +1$. Based on these definitions and assumptions, we can define the upper bound on the loss as follows:

$$\displaystyle \begin{aligned} F(\mathbf{w}[L]-F({\mathbf{w}}^*)) \leq \dfrac{1}{G\tau \left[A+B(1-C) \right]} {} \end{aligned} $$

(3.44)

It is observable that $A+B(1-C) \geq 0$, and further, the number of global updates G are fixed. Hence, the bound on the loss will converge to zero as $\tau \rightarrow \infty$.

Appendix 2

Let us write the KKT optimality conditions for (3.25) as shown in (3.45) and (3.51). The conditions (3.45) ensures that dataset size of any learner $k \in \mathcal {K}$ satisfies (3.27) and (3.47) ensures that the bound in (3.27) holds with equality for any learner $k \in \mathcal {K}$ if $\lambda _k^* \geq 0$. This is significant because strong duality holds for some feasible $\tau ^*$ when strictly speaking, $\lambda _k^* > 0 ~\forall ~k \in \mathcal {K}$. This means the upper bound will be the optimal solution.

$$\displaystyle \begin{gathered} C_k^2 \tau^* d_k^* + C_k^1 d_k^* + C_k^0 - T \leq 0, \quad k \in \mathcal{K} {} \end{gathered} $$

(3.45)

$$\displaystyle \begin{gathered} \alpha_0^*, \alpha_k^*, ~ \text{and} ~ \lambda_k^* \geq 0 \quad k \in \mathcal{K} \end{gathered} $$

(3.46)

$$\displaystyle \begin{gathered} \lambda_k^*\left(C_k^2 \tau^* d_k^* + C_k^1 d_k^* + C_k^0 - T\right) = 0, \quad k \in \mathcal{K} {} \end{gathered} $$

(3.47)

$$\displaystyle \begin{gathered} -\alpha_0^* \tau^* = 0 {} \end{gathered} $$

(3.48)

$$\displaystyle \begin{gathered} -\alpha_k^* d_k^* = 0 \quad k \in \mathcal{K} {} \end{gathered} $$

(3.49)

$$\displaystyle \begin{gathered} \sum_{k = 1}^{K}d_k^* - d = 0 {} \end{gathered} $$

(3.50)

$$\displaystyle \begin{aligned} -\nabla\tau^* + \sum_{k = 1}^{K} \lambda_k^*\nabla\left(C_k^2 \tau^* d_k^* + C_k^1 d_k^* + C_k^0 - T\right) + \\ \nu^*\nabla\left(\sum_{k = 1}^{K}d_k^* - d\right) - \alpha_0^*\nabla\tau^* - \nabla\left(\sum_{k = 1}^{K}\alpha_k^*d_k^*\right) = 0 {} \end{aligned} $$

(3.51)

We can then rewrite the bound on $d_k^*$ in (3.27) as an equality and substitute it back in (3.50) to obtain the following relation:

$$\displaystyle \begin{aligned} d= \sum_{k=1}^{K} d_k^* = \sum_{k=1}^{K} \left[ \dfrac{T-C_k^0}{\tau^* C_k^2+C_k^1} \right] = \sum_{k=1}^{K} \left[ \dfrac{a_k}{\tau^* + b_k} \right] {} \end{aligned} $$

(3.52)

The expression on the rightmost hand-side has the form of a partial fraction expansion of a rational polynomial function of $\tau ^*$ where $a_k, b_k \in \mathcal {R}^{++}$. Therefore, we can expand (3.52) to the form shown in (3.53).

$$\displaystyle \begin{aligned} \dfrac{a_1}{\tau^* + b_1} + \dfrac{a_2}{\tau^* + b_2} + \dots + \dfrac{a_k}{\tau^* + b_k} + \dots + \dfrac{a_K}{\tau^* + b_K} = \\ \dfrac{1}{(\tau^* + b_1)(\tau^* + b_2)\dots(\tau^* + b_k)\dots(\tau^* + b_K)} \times \\ \Bigg[ a_1(\tau^* + b_2)(\tau^* + b_3)\dots(\tau^* + b_k)\dots(\tau^* + b_K) + ~ \\ a_2(\tau^* + b_1)(\tau^* + b_3)\dots(\tau^* + b_k)\dots(\tau^* + b_K) + \ldots + \\ a_k(\tau^* + b_1)(\tau^* + b_2)\dots(\tau^* + b_{k-1})(\tau^* + b_{k+1})\dots(\tau^* + b_K) \\ + \dots + a_K(\tau^* + b_1)(\tau^* + b_2)\dots(\tau^* + b_k)\dots(\tau^* + b_{K-1}) \Bigg] {} \end{aligned} $$

(3.53)

Finally, the expanded form can be cleaned up in the form of a rational function with respect to $\tau ^*$, which is equal to the total dataset size d as shown in (3.54). Please note that the degrees of the numerator and denominator will be $K-1$ and K, respectively. Furthermore, the poles of the system will be $-b_k$, and since $b_k \geq 0$, the system will be stable. Furthermore, $\tau ^* = -b_k$ is not a feasible solution for the problem because it is eliminated by the $\tau \geq 0$ constraint. Therefore, we can rewrite (3.54) as shown in (3.28). By solving this polynomial, we obtain a set of solutions for $\tau ^*$, where one of them is feasible. The problem being non-convex, this feasible solution $\tau ^*$ will constitute the upper bound to the solution of the relaxed problem.

$$\displaystyle \begin{aligned} d = \dfrac{\sum_{k=1}^{K} a_k \prod_{\substack{l=1 \\ l\neq k}}^{K} \left(\tau^*+b_l\right)}{\prod_{k=1}^{K} \left(\tau^*+b_k\right)} {} \end{aligned} $$

(3.54)

As a last step, to ensure that the solution set is feasible, it must be noted that according to (3.48) and (3.49), $\alpha _0^*$ and $\alpha _k^* ~ \forall ~ k$ must be equal to 0. Expanding the vanishing gradient condition in (3.51), it can be shown that the following two relations can be derived (representing $K+1$ equations):

$$\displaystyle \begin{aligned} \lambda_k^*C_k^2\tau^* + \lambda_k^*C_k^1 + \nu^* = \alpha_k^*, ~ k \in \mathcal{K} {} \end{aligned} $$

(3.55)

$$\displaystyle \begin{aligned} -1+\sum_{k=1}^{K} \lambda_k^*C_k^2d_k^* = \alpha_0^* {} \end{aligned} $$

(3.56)

By setting $\alpha _0^* = 0$ and $\alpha _k^* = 0$ for $k \in \mathcal {K}$, we can write $\lambda _k^*$ in terms of $\nu ^*$ as shown in (3.57) and substitute the resulting expression in (3.56) to find $\nu ^*$ using the values of $d_k^*$ and $\tau ^*$ obtained from (3.27) and (3.28), respectively.

$$\displaystyle \begin{aligned} \lambda_k^* = -\dfrac{\nu^*}{C_k^2\tau^* + C_k^1}, ~ k \in \mathcal{K} {} \end{aligned} $$

(3.57)

$$\displaystyle \begin{aligned} \nu^* = -\dfrac{1}{\sum_{k=1}^{K} \frac{C_k^2d_k^*}{C_k^2\tau^* + C_k^1}} {} \end{aligned} $$

(3.58)

The values of $\lambda _k^*$ for $k \in \mathcal {K}$ can be obtained by back-substitution of $\nu ^*$ in (3.57). As one can observe, as long as there exists a $\tau ^*$ greater than zero, $\nu ^*$ will be negative and hence, $\lambda _k^*$ for $k \in \mathcal {K}$ will be strictly greater than zero. Hence, as long as there exists a $\tau ^* > 0$ in the feasible set such that $d_k^* > 0$, there will exist a set of $\lambda _k^* > 0$ for $k \in \mathcal {K}$. This fact can be used to verify the feasibility of the solution. This step is also helpful when there may exist multiple values of $\tau$ greater than zero for choosing the optimal $\tau ^*$. Extensive simulations presented in Sect. 3.5 demonstrated that there was no optimality gap between the analytical upper bounds and the numerical solution.

Appendix 3

Recall that the optimization variables are given by $\mathbf {x} = \left [\tau ~ d_1 ~ d_2 ~ \ldots ~ d_k ~ \ldots ~ d_K\right ]^T$. Then, the relaxed problem in (3.25) can be rewritten in the standard form of a QCQP as follows:

(3.59)

$$\displaystyle \begin{aligned} \text{s.t. }\qquad & {\mathbf{x}}^T {\mathbf{P}}_k \mathbf{x} +{\mathbf{p}}_k^T \mathbf{x} + p_k^0 \leq 0, \quad \forall k \in \mathcal{K} {} \end{aligned} $$

(3.59a)

$$\displaystyle \begin{aligned} & {\mathbf{x}}^T {\mathbf{Q}}_k \mathbf{x} +{\mathbf{q}}_k^T \mathbf{x} + q_k^0 \leq 0, \quad \forall k \in \mathcal{K} {} \end{aligned} $$

(3.59b)

$$\displaystyle \begin{aligned} & {\mathbf{x}}^T \mathbf{A} \mathbf{x} +{\mathbf{a}}^T \mathbf{x} + a_0 \leq 0 {} \end{aligned} $$

(3.59c)

$$\displaystyle \begin{aligned} & {\mathbf{x}}^T \bar{\mathbf{A}} \mathbf{x} +\bar{\mathbf{a}}^T \mathbf{x} + \bar{a}_0 \leq 0 {} \end{aligned} $$

(3.59d)

$$\displaystyle \begin{aligned} & {\mathbf{x}}^T \mathbf{U} \mathbf{x} +{\mathbf{U}}^T \mathbf{x} + u_0 \leq 0 {} \end{aligned} $$

(3.59e)

$$\displaystyle \begin{aligned} & {\mathbf{x}}^T {\mathbf{V}}_k \mathbf{x} +{\mathbf{v}}_k^T \mathbf{x} + v_k^0 \leq 0, \quad \forall k \in \mathcal{K} {} \end{aligned} $$

(3.59f)

The time and energy constraints are defined by (3.59a) and (3.59b), respectively. Constraints (3.59c) and (3.59d) are two inequality constraints used to simplify the equality constraint of total dataset size allocation. The nonnegative constraints on $\tau$ and $d_k$ are given in (3.59e) and (3.59f), respectively. The expressions for problem definition and each constraint have three terms each: a quadratic term, a linear term, and a constant term. The constant terms$p_k^0 = C_k^0-T$ and $q_k^0 = G_k^0-E_k^0 ~\forall ~ k \in \mathcal {K}$ and are associated with the time and energy constraints, respectively. The remaining constant terms $a_0 = -d$, $\bar {a}_o = d$ and $v_k^0 = d_l, \forall ~ k$ whereas $u_0 = 0$ and $f_0 =0$. In contrast, the coefficients associated with the linear terms in the objective ($\mathbf {f}$) and constraints ($~{\mathbf {p}}_k$, ${\mathbf {q}}_k$, $\mathbf {a}$, $\bar {\mathbf {a}}$, $\mathbf {u}$, and ${\mathbf {v}}_k$) can be represented by the following set of vectors:

$$\displaystyle \begin{aligned} &\quad \mathbf{f} = \left[-1 ~ 0 ~ 0 ~ \ldots ~ C_k^1 ~ \ldots ~ 0\right]^T \\ &\quad {\mathbf{p}}_k = \left[0 ~ 0 ~ 0 ~ \ldots ~ C_k^1 ~ \ldots ~ 0\right]^T, \forall ~ k\\ &\quad {\mathbf{q}}_k = \left[0 ~ 0 ~ 0 ~ \ldots ~ G_k^1 ~ \ldots ~ 0\right]^T, \forall ~ k\\ &\quad \mathbf{a} = \left[0 ~ 1 ~ 1 ~ \ldots ~ 1 ~ \ldots ~ 1\right]^T\\ &\quad \bar{\mathbf{a}} = \left[0 ~ -1 ~ -1 ~ \ldots ~ -1 ~ \ldots ~ -1\right]^T\\ &\quad \mathbf{u} = \left[-1 ~ 0 ~ 0 ~ \ldots ~ 0 ~ \ldots ~ 0\right]^T\\\ &\quad {\mathbf{v}}_k = \left[0 ~ 0 ~ 0 ~ \ldots ~ -1 ~ \ldots ~ 0\right]^T, \forall ~ k \end{aligned} $$

(3.60)

In general, the coefficients associated with the quadratic terms are $(K+1) \times (K+1)$ matrices. Because this is a QCLP, the quadratic term in the objective is ${\mathbf {0}}_{(K+1) \times (K+1)}$, a $(K+1) \times (K+1)$ zero matrix. The coefficients associated with the time and energy constraints, ${\mathbf {P}}_k$ and ${\mathbf {Q}}_k$, respectively, can be described as follows:

$$\displaystyle \begin{aligned} {\mathbf{P}}_k(i,j) = \begin{cases} 0.5C_k^2, \mbox{ if} & \begin{matrix} i = 1 ~ \& ~ j = k+1 \\ i = k+1 ~ \& ~ j = 1 \end{matrix}\\ 0, & \mbox{otherwise} \end{cases} {} \end{aligned} $$

(3.61)

$$\displaystyle \begin{aligned} {\mathbf{Q}}_k(i,j) = \begin{cases} 0.5G_k^2, \mbox{ if} & \begin{matrix} i = 1 ~ \& ~ j = k+1 \\ i = k+1 ~ \& ~ j = 1 \end{matrix}\\ 0, & \mbox{otherwise} \end{cases} {} \end{aligned} $$

(3.62)

The quadratic coefficients of the remaining constraints $\mathbf {A}$, $\bar {\mathbf {A}}$, $\mathbf {U}$ and ${\mathbf {V}}_k$ are all ${\mathbf {0}}_{(K+1) \times (K+1)}$. We can now define the functions ${\mathbf {F}}^2(\boldsymbol {\Gamma })$, ${\mathbf {f}}^1(\boldsymbol {\Gamma })$ and $f_0(\boldsymbol {\Gamma })$ as [26]:

$$\displaystyle \begin{aligned} {\mathbf{F}}^2(\boldsymbol{\Gamma}) = \sum_{k = 1}^{K} \lambda_k {\mathbf{P}}_k + \gamma_k {\mathbf{Q}}_k {} \end{aligned} $$

(3.63)

$$\displaystyle \begin{aligned} {\mathbf{f}}^1(\boldsymbol{\Gamma}) = \sum_{k = 1}^{K}\left( \lambda_k {\mathbf{p}}_k + \gamma_k {\mathbf{q}}_k + \nu_k {\mathbf{v}}_k\right) + \alpha \mathbf{a} + \bar{\alpha} \bar{\mathbf{a}} + \omega \mathbf{u} {} \end{aligned} $$

(3.64)

$$\displaystyle \begin{aligned} f_0(\boldsymbol{\Gamma}) = \sum_{k = 1}^{K}\left( \lambda_k p_k^0 + \gamma_k q_k^0 + \nu_k v_k^0 \right) + \alpha a_0 + \bar{\alpha} \bar{a}_0 {} \end{aligned} $$

(3.65)