Generalization in Deep RL for TSP Problems via Equivariance and Local Search

Abstract

Deep reinforcement learning (RL) has proved to be a competitive heuristic for solving small-sized instances of traveling salesman problems (TSP), but its performance on larger-sized instances is insufficient. Since training on large instances is impractical, we design a novel deep RL approach with a focus on generalizability. Our proposition consisting of a simple deep learning architecture, which learns with novel RL training techniques, exploits two main ideas. First, we exploit equivariance to facilitate training. Second, we interleave efficient local search heuristics with the usual RL training to smooth the value landscape. Our experimental evaluation demonstrates that our method achieves state-of-the-art performances not only when generalizing to large random TSP instances, but also on realistic TSP instances. Moreover, an ablation study shows that all the components of our method contribute to its performance.

Appendices

Appendix A: Policy Gradient of eMAGIC

As promised in Sect. “Algorithm & Training” - Smoothed Policy Gradient, we elaborate on the detailed mathematical derivation for Equation (13).

The objective function $J(\varvec{\theta })$ in Eq. (3) can be approximated with the empirical mean of the total rewards using B trajectories sampled with policy $\pi _{\varvec{\theta }}$:

$$\begin{aligned} J(\varvec{\theta }) = {-{\mathbb {E}}[L_\sigma ({\varvec{X}})]}\approx {-\hat{{\mathbb {E}}}_B\left[ \sum _{t=1}^N r^{(b)}_{t}\right] } = {-\frac{1}{|B|} \sum _{b=1}^{|B|} \sum _{t=1}^{N} r^{(b)}_{t}}, \end{aligned}$$

(16)

where $\hat{{\mathbb {E}}}_B$ represents the empirical mean operator, $L_\sigma ({\varvec{X}})$ is the tour length of $\sigma$ output by the RL policy, and $r^{(b)}_{t}$ is the t-th reward of the b-th trajectory. The policy gradient to optimize $J(\varvec{\theta })$ can be estimated by:

$$\begin{aligned} \begin{aligned} \nabla _{\varvec{\theta }} J(\varvec{\theta })&=-{\mathbb {E}}_\tau \left[ \left( \sum _{t=1}^{N} \nabla _{\varvec{\theta }} \log \pi _{\varvec{\theta }}({a}_{t} \mid \varvec{s}_{t})\right) \left( \sum _{t=1}^{N} r_t\right) \right] \\ {}&\approx -\hat{\mathbb E}_B\left[ \left( \sum _{t=1}^{N} \nabla _{\varvec{\theta }} \log \pi _{\varvec{\theta }}({a}^{(b)}_{t} \mid \varvec{s}^{(b)}_{t})\right) \left( \sum _{t=1}^{N} r^{(b)}_{t}\right) \right] \end{aligned} \end{aligned}$$

(17)

However, recall $J(\varvec{\theta })$ is the standard objective used in most deep RL methods applied to TSP. Instead, we optimize $J^+(\varvec{\theta })= -{\mathbb {E}}[L_{\sigma _+}({\varvec{X}})]$ where $L_{\sigma _+}({\varvec{X}})$ is the tour length of $\sigma$ after applying local search. This helps integrate better RL and local search by smoothing the value landscape and training an RL agent to output a tour that can be improved by local search. This new objective function can be rewritten:

$$\begin{aligned} \begin{aligned} J^+(\varvec{\theta })&= -{\mathbb {E}}_{\sigma \sim \pi _{\varvec{\theta }},\sigma _+\sim \rho (\sigma )}[L_{\sigma _+}({\varvec{X}})]\\&= -{\mathbb {E}}_{\sigma \sim \pi _{\varvec{\theta }}}[\mathbb E_{\sigma _+\sim \rho (\sigma )}[L_{\sigma _+}({\varvec{X}}) \mid \sigma ]] \end{aligned} \end{aligned}$$

(18)

where $\rho (\sigma )$ denotes the distribution over tours induced by the application of the stochastic local search on $\sigma$. Taking the gradient of this new objective:

$$\begin{aligned} \begin{aligned}&\nabla _{\varvec{\theta }} J^+(\varvec{\theta })= - {\mathbb {E}}_\tau \left[ \left( \sum _{t=1}^{N} \nabla _{\varvec{\theta }} \log \pi _{\varvec{\theta }}({a}_{t} \mid {\varvec{s}}_{t})\right) {\mathbb {E}}_{\sigma _+\sim \rho (\sigma )}[L_{\sigma _+}({\varvec{X}}) \mid \sigma ] \right] \\&\approx - \hat{{\mathbb {E}}}_B\left[ \left( \sum _{t=1}^{N} \nabla _{\varvec{\theta }} \log \pi _{\varvec{\theta }}({a}^{(b)}_{t} \mid {\varvec{s}}^{(b)}_{t})\right) \left( L_{\sigma ^{(b)}_+}({\varvec{X}}^{(b)})\right) \right] \\&\approx - \frac{1}{|B|} \sum _{b=1}^{|B|} \left( \sum _{t=2}^{N} \nabla _{\varvec{\theta }} \log \pi _{\varvec{\theta }}({a}^{(b)}_{t} \mid \varvec{s}^{(b)}_{t})\right) \left( L_{\sigma _+^{(b)}}(\varvec{X}^{(b)})-l^{(b)}\right) , \end{aligned} \end{aligned}$$

(19)

where $\tau = ({\varvec{s}}_1, a_1, \ldots )$ and $\sigma$ is its associated tour. We simply approximate the conditional expectation over $\rho (\sigma )$ by a sample. Therefore, our gradient estimate is an unbiased estimator of the gradient of our new objective $J^+(\varvec{\theta })$.

Using our policy rollout baseline introduces some bias to the estimation of the smoothed policy gradient. However, the variance reduction helps with achieving greater performance and smaller variance, as we observed in our experiments.

Appendix B: Settings and Hyperparameters

All our experiments are run on a computer with an Intel(R) Xeon(R) E5-2678 v3 CPU and a NVIDIA 1080Ti GPU. In consistency with previous work, all our randomly generated TSP instances are sampled in $[0,1]^2$ uniformly. During training, stochastic CL chooses a TSP size in ${\mathcal {R}}=\{10,11,\dots ,50\}$ for each epoch e according to Eqs. (14) and (15), with $\sigma _N$ set to be 3. We trained for 200 epochs in each experiment, with 1000 batches of 128 random TSP instances in each epoch. We set the learning rate to be $10^{-3}$ and the learning rate decay to be 0.96 in each experiment. In each experiment, we set $\alpha =0.5$, $\beta =1.5$, $\gamma =0.25$ and $I=10$ for the parameters of our combined local search. As for random TSP testing and the ablation study, we test on TSP instances with size 20, 50, 100, 200, 500, and 1000 to evaluate the generalization capability of our model; as for realistic TSP, we test on TSP instances with sizes up to 1002 from the TSPLIB library. With respect to our model architecture, our MLP encoder has an input layer with dimension 2, two hidden layers with dimension 128 and 256, respectively, and an output layers with dimension 128; for the GNN encoder, we set $H=128$ and $n_{GNN}=3$.

Addendix C: More Experiments on eMAGIC

More versions of eMAGIC

As promised in Sect. 6, we illustrate all versions of eMAGIC in this section. See Tables 7 and 8.

Table 7 Results of eMAGIC vs baselines, tested on 10,000 instances for TSP 20, 50, and 100

Table 8 Results of all versions of eMAGIC, tested on 128 instances for TSP 200, 500, and 1000

Variance Analysis of eMAGIC

As promised in Sect. 6, we provide the variance analysis of eMAGIC in this section. In our experiments, we repeated all our experiments (training + testing) with 3 random seeds. The variances are shown in Table 9.

Table 9 Variance analysis of eMAGIC

The variances are quite low, showing that our method gives relatively good and stable results. Note the variances increase with the number of sampling (s1, s10 and S) for larger TSP instances since there is more room for improvement.

Experiments on Extremely Large TSP Instances

In Table 10, we show the performances of eMAGIC on extremely large TSP instances (i.e., TSP 10,000) and the comparisons with a few methods that are able to generalize to TSP 10,000. For the hyperparameters of the combined local search, we use $I=2$ and $\beta =1.4$. We modify the hyperparameters in this way because we find that for large TSP instances, doing too many iterations of local search is not efficient. The other experimental settings in this test are the same as Sect. 6. Also, we test [12]’s method with a limited the time budget, denoted by AGCRN+M$^{lim}$: Since we only reproduce their method successfully with CPU, we decrease their hyperparameter T (the MCTS will end no longer than T seconds) from 0.04n to $0.04n / 1.8 * (28/60) = 0.010n$ (1.8h and 28 m are, respectively, the runtimes of their model and eMAGIC(G)). By this, we expect that their model and eMAGIC(G) will have a similar running time.

Table 10 Results of eMAGIC vs baselines, tested on 16 instances for TSP 10,000

We can observe that our models outperform AM [20] to a large extent. Comparing to AGCRN+M [12] and LKH3 [15], our methods run much faster without a big gap of performance. Plus, if we give AGCRN+M [12] a time budget comparable to our method (i.e., run AGCRN+M and eMAGIC(G) for the same amount of time), we can see that our algorithm outperforms AGCRN+M. Note that MCTS in AGCRN+M is written in C++. We could reduce our runtime if we had also written our code in C++ instead of Python. Therefore, our method offers a better trade-off in terms of performance vs runtime: It can generate relatively good results in much less time.

Appendix D: Evaluation of Equivariant Preprocessing Methods

Comparisons Between Different Symmetries Used During Preprocessing

We present the comparison results of applying rotation, translation and reflection. The comparisons are done by testing on random TSP instances followed the same setting in the experiment section. As in Table 11, the rotation has the best performance on small and large TSP instances. By this, we choose rotation for our final algorithm.

Table 11 Comparisons between rotation, translation, and reflection

Comparisons Between One Preprocessing Application and Iteration Preprocessing Applications

We present the comparison results of one preprocessing application and iteration preprocessing applications. The comparisons are done by testing on random TSP instances followed the same setting in the experiment section. As in Table 12, the iteration preprocessing applications have better performance on small and large TSP instances. By this, we choose iteration preprocessing applications for our final algorithm.

Table 12 Comparisons between iteration preprocessing and one preprocessing

Realist TSP Instances in TSPLIB

As promised in Sect. 6, Tables 13, 14 and 15 list the performances of various learning-based TSP solvers and heuristic approaches upon realistic TSP instances in TSPLIB. The bold numbers show the best performance among all the approaches. We can observe that most bold numbers are provided by eMAGIC, meaning our approach provides excellent results for TSPLIB. In addition, we also provide our results for the larger instances (with size up to 1002) from TSPLIB in Table 16. Table 16 only contains the experiments of our model since no other model can generalize to large size TSP instances with adequate performance.

Table 13 Comparison of Performances on TSPLIB—Part I

Table 14 Comparison of performances on TSPLIB—Part II

Table 15 Comparison of Performances on TSPLIB—Part III

Table 16 Comparison of performance on large size TSP instances in TSPLIB