Abstract
Combinatorial optimization (CO) on graphs is a classic topic that has been extensively studied across many scientific and industrial fields. Recently, solving CO problems on graphs through learning methods has attracted great attention. Advanced deep learning methods, e.g., graph neural networks (GNNs), have been used to effectively assist the process of solving COs. However, current frameworks based on GNNs are mainly designed for certain CO problems, thereby failing to consider their transferable and generalizable abilities among different COs on graphs. Moreover, simply using original graphs to model COs only captures the direct correlations among objects, which does not consider the mathematical logicality and properties of COs. In this paper, we propose a unified pre-training and adaptation framework for COs on graphs with the help of the maximum satisfiability (Max-SAT) problem. We first use Max-SAT to bridge different COs on graphs since they can be converted to Max-SAT problems represented by standard formulas and clauses with logical information. Then we further design a pre-training and domain adaptation framework to extract the transferable and generalizable features so that different COs can benefit from them. In the pre-training stage, Max-SAT instances are generated to initialize the parameters of the model. In the fine-tuning stage, instances from CO and Max-SAT problems are used for adaptation so that the transferable ability can be further improved. Numerical experiments on several datasets show that features extracted by our framework exhibit superior transferability and Max-SAT can boost the ability to solve COs on graphs.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Aarts E H L, Korst J H M, van Laarhoven P J M. A quantitative analysis of the simulated annealing algorithm: A case study for the traveling salesman problem. J Stat Phys, 1988, 50: 187–206
Agarwal C, Queen O, Lakkaraju H, et al. Evaluating explainability for graph neural networks. Scientific Data, 2023, 10: 144
Andrade D V, Resende M G C, Werneck R F. Fast local search for the maximum independent set problem. J Heuristics, 2012, 18: 525–547
Ansótegui C, Bonet M L, Levy J. Towards industrial-like random sat instances. In: Proceedings of the 21st International Joint Conference on Artificial Intelligence. Freiburg: Inst Inform, 2009, 387–392
Barrett T, Clements W, Foerster J, et al. Exploratory combinatorial optimization with reinforcement learning. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34. Palo Alto: AAAI Press, 2020, 3243–3250
Bellman R. Dynamic programming. Science, 1966, 153: 34–37
Benlic U, Hao J-K. Breakout local search for the Max-Cut problem. Engrg Appl Artificial Intell, 2013, 26: 1162–1173
Benson S J, Ye Y Y, Zhang X. Solving large-scale sparse semidefinite programs for combinatorial optimization. SIAM J Optim, 2000, 10: 443–461
Berger J, Barkaoui M. A hybrid genetic algorithm for the capacitated vehicle routing problem. In: Genetic and Evolutionary Computation Conference. New York: Springer, 2003, 646–656
Bertsimas D, Demir R. An approximate dynamic programming approach to multidimensional knapsack problems. Management Sci, 2002, 48: 550–565
Bodnar C, Di Giovanni F, Chamberlain B, et al. Neural sheaf diffusion: A topological perspective on heterophily and oversmoothing in GNNs. Adv Neural Inform Proc Syst, 2022, 35: 18527–18541
Boettcher S, Percus A G. Extremal optimization for graph partitioning. Phys Rev E, 2001, 64: 026114
Bouman P, Agatz N, Schmidt M. Dynamic programming approaches for the traveling salesman problem with drone. Networks, 2018, 72: 528–542
Cappart Q, Chételat D, Khalil E B, et al. Combinatorial optimization and reasoning with graph neural networks. J Mach Learn Res, 2023, 24: 1–61
Chassein A, Goerigk M, Kasperski A, et al. Approximating combinatorial optimization problems with the ordered weighted averaging criterion. European J Oper Res, 2020, 286: 828–838
Chen T L, Chen X H, Chen W Y, et al. Learning to optimize: A primer and a benchmark. J Mach Learn Res, 2022, 23: 8562–8620
Choi C H, Ye Y Y. Solving sparse semidefinite programs using the dual scaling algorithm with an iterative solver. Working paper, https://www.mcs.anl.gov/research/projects/otc/InteriorPoint/abstracts/Choi-Ye-2.html, 2000
Dai Q Y, Wu X-M, **ao J R, et al. Graph transfer learning via adversarial domain adaptation with graph convolution. IEEE Trans Knowledge Data Eng, 2022, 35: 4908–4922
Dembo A, Montanari A, Sen S. Extremal cuts of sparse random graphs. Ann Probab, 2017, 45: 1190–1217
Di Giovanni F, Giusti L, Barbero F, et al. On over-squashing in message passing neural networks: The impact of width, depth, and topology. In: International Conference on Machine Learning. New York: PMLR, 2023, 7865–7885
Feo T A, Resende M G C, Smith S H. A greedy randomized adaptive search procedure for maximum independent set. Oper Res, 1994, 42: 860–878
Fischetti M, Toth P, Vigo D. A branch-and-bound algorithm for the capacitated vehicle routing problem on directed graphs. Oper Res, 1994, 42: 846–859
Gasse M, Chételat D, Ferroni N, et al. Exact Combinatorial Optimization with Graph Convolutional Neural Networks. Advances in Neural Information Processing Systems, vol. 32. La Jolla: NIPS, 2019
Gilmer J, Schoenholz S S, Riley P F, et al. Neural message passing for quantum chemistry. In: International Conference on Machine Learning. New York: PMLR, 2017, 1263–1272
Hamilton W, Ying Z T, Leskovec J. Inductive Representation Learning on Large Graphs. Advances in Neural Information Processing Systems, vol. 30. La Jolla: NIPS, 2017
Held M, Karp R M. A dynamic programming approach to sequencing problems. J Soc Indust Appl Math, 1962, 10: 196–210
Khalil E, Dai H J, Zhang Y Y, et al. Learning Combinatorial Optimization Algorithms over Graphs. Advances in Neural Information Processing Systems, vol. 30. La Jolla: NIPS, 2017
Kim M, Park J, Park J. SYM-NCO: Leveraging symmetricity for neural combinatorial optimization. Adv Neural Inform Proc Syst, 2022, 35: 1936–1949
Kipf T N, Welling M. Semi-supervised classification with graph convolutional networks. ar**v:1609.02907, 2016
Kool W, van Hoof H, Welling M. Attention, learn to solve routing problems! ar**v:1803.08475, 2018
Krislock N, Malick J, Roupin F. Improved semidefinite bounding procedure for solving Max-Cut problems to optimality. Math Program, 2014, 143: 61–86
Lawler E L, Wood D E. Branch-and-bound methods: A survey. Oper Res, 1966, 14: 699–719
Lemos H, Prates M, Avelar P, et al. Graph coloring meets deep learning: Effective graph neural network models for combinatorial problems. In: 2019 IEEE 31st International Conference on Tools with Artificial Intelligence (ICTAI). New York: IEEE, 2019, 879–885
Li Z W, Chen Q F, Koltun V. Combinatorial Optimization with Graph Convolutional Networks and Guided Tree Search. Neural Information Processing Systems, vol. 31. La Jolla: NIPS, 2018
Liu M H, Huang P, Jia F Q, Zhang F, et al. Can graph neural networks learn to solve the maxsat problem? In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37. Palo Alto: AAAI Press, 2023, 16264–16265
Liu S K, Li T C, Feng Y B, et al. Structural re-weighting improves graph domain adaptation. In: International Conference on Machine Learning. New York: PMLR, 2023, 21778–21793
Los M, Lardinois C. Combinatorial programming, statistical optimization and the optimal transportation network problem. Transp Res Part B Methodological, 1982, 16: 89–124
Ma X H, Zhang T Z, Xu C S. Gcan: Graph convolutional adversarial network for unsupervised domain adaptation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2019, 8266–8276
Martello S, Pisinger D, Toth P. Dynamic programming and strong bounds for the 0–1 knapsack problem. Management Sci, 1999, 45: 414–424
Martins S L, Ribeiro C C. Metaheuristics and applications to optimization problems in telecommunications. In: Handbook of Optimization in Telecommunications. Boston: Springer, 2006, 103–128
Nguyen K, Hieu N M, Nguyen V D, et al. Revisiting over-smoothing and over-squashing using Ollivier-Ricci curvature. In: International Conference on Machine Learning. New York: PMLR, 2023, 25956–25979
Orlin J B, Punnen A P, Schulz A S. Approximate local search in combinatorial optimization. SIAM J Comput, 2004, 33: 1201–1214
Pardalos P M, Rodgers G P. A branch and bound algorithm for the maximum clique problem. Comput Oper Res, 1992, 19: 363–375
Pilanci M, Vural E. Domain adaptation on graphs by learning aligned graph bases. IEEE Trans Knowledge Data Eng, 2022, 34: 587–600
Pop P C, Cosma O, Sabo C, et al. A comprehensive survey on the generalized traveling salesman problem. European J Oper Res, 2024, 314: 819–835
Schuetz M J A, Brubaker J K, Katzgraber H G. Combinatorial optimization with physics-inspired graph neural networks. Nature Mach Intell, 2022, 4: 367–377
Sun M, Zhou K X, He X, et al. GPPT: Graph pre-training and prompt tuning to generalize graph neural networks. In: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. New York: ACM, 2022, 1717–1727
Toenshoff J, Ritzert M, Wolf H, et al. Graph neural networks for maximum constraint satisfaction. Front Artificial Intell, 2021, 3: 580607
Veličković P, Cucurull G, Casanova A, et al. Graph Attention Networks. Proceedings of the 6th International Conference on Learning Representations. Vancouver: OpenReview.net, 2018
Wang Y Y, Cai S W, Pan S W, et al. Reduction and local search for weighted graph coloring problem. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34. Palo Alto: AAAI Press, 2020, 2433–2441
Wolsey L A, Nemhauser G L. Integer and Combinatorial Optimization. New York: John Wiley & Sons, 1999
Xu K, Hu W H, Leskovec J, et al. How powerful are graph neural networks? ar**v:1810.00826, 2018
Yagiura M, Ibaraki T. On metaheuristic algorithms for combinatorial optimization problems. Syst Comput Japan, 2001, 32: 33–55
Yao W C, Bandeira A S, Villar S. Experimental performance of graph neural networks on random instances of Max-Cut. In: Wavelets and Sparsity XVIII, vol. 11138. Bellingham: SPIE, 2019, 242–251
Yi Z X, Ounis I, Macdonald C. Contrastive Graph Prompt-Tuning for Cross-Domain Recommendation. ACM Transactions on Information Systems. New York: ACM, 2023
You Y N, Chen T L, Wang Z Y, et al. Graph domain adaptation via theory-grounded spectral regularization. In: International Conference on Learning Representations, https://openreview.net/pdf?id=OysfLgrk8mk, 2023
Yuan J, Hou F, Du Y Z, et al. Self-supervised graph neural network for multi-source domain adaptation. In: Proceedings of the 30th ACM International Conference on Multimedia. New York: ACM, 2022, 3907–3916
Zhang B H, Luo S J, Wang L W, et al. Rethinking the expressive power of GNNs via graph biconnectivity. ar**v:2301.09505, 2023
Zhu R H, Jiang X D, Lu J S, et al. Cross-domain graph convolutions for adversarial unsupervised domain adaptation. IEEE Trans Neural Netw Learn Syst, 2023, 34: 3847–3858
Acknowledgements
This work was supported by National Natural Science Foundation of China (Grant Nos. 11991021, 11991020 and 12271503).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Zeng, R., Lei, M., Niu, L. et al. A unified pre-training and adaptation framework for combinatorial optimization on graphs. Sci. China Math. 67, 1439–1456 (2024). https://doi.org/10.1007/s11425-023-2247-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11425-023-2247-0