Abstract
Electrical and Electronic Equipment (EEE) has evolved into a gateway for accessing technological innovations. However, EEE imposes substantial pressure on the environment due to the shortened life cycles. E-waste encompasses discarded EEE and its components which are no longer in use. This study focuses on the e-waste collection problem and models it as a Vehicle Routing Problem with a heterogeneous fleet and a multi-period planning problem with time windows as well as stochastic travel times. Two different Q-learning-based methods are designed to enhance the search procedure for finding solutions. The first method involves utilizing the state-action value to determine the order of multiple improvement operators within the GRASP framework. The second one involves a hyperheuristic that extracts a stochastic policy to select heuristic operators during the search. Computational experiments demonstrate that both methods perform competitively with state-of-the-art methods in newly-generated small-sized instances, while the performance gap widens as the size of the problem instances increases.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Pérez-Belis, V., Bovea, M.D., Ibáñez-Forés, V.: An in-depth literature review of the waste electrical and electronic equipment context: trends and evolution. Waste Manage. Res. 33(1), 3–29 (2015)
Wu, H., Tao, F., Yang, B.: Optimization of vehicle routing for waste collection and transportation. Int. J. Environ. Res. Public Health 17(14), 4963 (2020)
Szwarc, K., Nowakowski, P., Boryczka, U.: An evolutionary approach to the vehicle route planning in e-waste mobile collection on demand. Soft. Comput. 25(8), 6665–6680 (2021)
Pourhejazy, P., Zhang, D., Zhu, Q., Wei, F., Song, S.: Integrated e-waste transportation using capacitated general routing problem with time-window. Transp. Res. Part E: Logist. Transp. Rev. 145, 102169 (2021)
Gunawan, A., Nguyen, M.P.K., Vincent, F.Y., Nguyen, D.V.A.: The heterogeneous vehicle routing problem with multiple time windows for the e-waste collection problem. In: 19th International Conference on Automation Science and Engineering (CASE) (2023)
Gunawan, A., Nguyen, D.V.A., Nguyen, P.K.M., Vansteenwegen, P.: Grasp solution approach for the e-waste collection problem. In: Daduna, J.R., Liedtke, G., Shi, X., Voß, S. (eds.) ICCL 2023. LNCS, vol. 14239, pp. 260–275. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-43612-3_16
Król, A., Nowakowski, P., Mrówczyńska, B.: How to improve WEEE management? Novel approach in mobile collection with application of artificial intelligence. Waste Manage. 50, 222–233 (2016)
Karimi-Mamaghan, M., Mohammadi, M., Meyer, P., Karimi-Mamaghan, A.M., Talbi, E.G.: Machine learning at the service of meta-heuristics for solving combinatorial optimization problems: a state-of-the-art. Eur. J. Oper. Res. 296(2), 393–422 (2022)
Talbi, E.G.: Machine learning into metaheuristics: a survey and taxonomy. ACM Comput. Surv. (CSUR) 54(6), 1–32 (2021)
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (2018)
Lu, H., Zhang, X., Yang, S.: A learning-based iterative method for solving vehicle routing problems. In: International Conference on Learning Representations (2019)
Peng, B., Zhang, Y., Gajpal, Y., Chen, X.: A memetic algorithm for the green vehicle routing problem. Sustainability 11(21), 6055 (2019)
Reijnen, R., Zhang, Y., Lau, H.C., Bukhsh, Z.: Operator selection in adaptive large neighborhood search using deep reinforcement learning. ar**v preprint ar**v:2211.00759 (2022)
Ödling, D.: A metaheuristic for vehicle routing problems based on reinforcement learning (2018)
Watkins, C.J., Dayan, P.: Q-learning. Mach. Learn. 8, 279–292 (1992)
Festa, P., Pastore, T., Ferone, D., Juan, A.A., Bayliss, C.: Integrating biased-randomized GRASP with monte carlo simulation for solving the vehicle routing problem with stochastic demands. In: 2018 Winter Simulation Conference (WSC), pp. 2989–3000. IEEE (2018)
Golden, B., Assad, A., Levy, L., Gheysens, F.: The fleet size and mix vehicle routing problem. Comput. Oper. Res. 11(1), 49–66 (1984)
Kallestad, J., Hasibi, R., Hemmati, A., Sörensen, K.: A general deep reinforcement learning hyperheuristic framework for solving combinatorial optimization problems. Eur. J. Oper. Res. 309(1), 446–468 (2023)
Sutton, R.S., McAllester, D., Singh, S., Mansour, Y.: Policy gradient methods for reinforcement learning with function approximation. In: Advances in Neural Information Processing Systems, vol. 12 (1999)
Konda, V., Tsitsiklis, J.: Actor-critic algorithms. In: Advances in Neural Information Processing Systems, vol. 12 (1999)
Schulman, J., Levine, S., Abbeel, P., Jordan, M., Moritz, P.: Trust region policy optimization. In: International Conference on Machine Learning, pp. 1889–1897. PMLR (2015)
Sheskin, D.J.: Handbook of Parametric and Nonparametric Statistical Procedures. CRC Press, Boca Raton (2020)
Acknowledgement
This research was supported by the Singapore Ministry of Education (MOE) Academic Research Fund (AcRF) Tier 1 grant.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Nguyen, D.V.A., Gunawan, A., Misir, M., Vansteenwegen, P. (2024). Q-Learning Based Framework for Solving the Stochastic E-waste Collection Problem. In: Stützle, T., Wagner, M. (eds) Evolutionary Computation in Combinatorial Optimization. EvoCOP 2024. Lecture Notes in Computer Science, vol 14632. Springer, Cham. https://doi.org/10.1007/978-3-031-57712-3_4
Download citation
DOI: https://doi.org/10.1007/978-3-031-57712-3_4
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-57711-6
Online ISBN: 978-3-031-57712-3
eBook Packages: Computer ScienceComputer Science (R0)