Abstract
Traditionally, adaptive instructional systems (AISs) are built to instruct human students. However, they are not the only students that might benefit from an AIS. The field of reinforcement learning (RL), a subfield of machine learning, studies the instruction of synthetic students called agents, by means of various algorithms. In this paper, we advocate the use of an AIS as a conceptual framework to design and teach RL agents. We form our argument by deconstructing what it means to build and use an AIS for a human student, and discuss how the various concepts and relationships may apply to RL agents. We illustrate our findings by means of examples from the reinforcement learning literature and show a domain implementation of an AIS for RL agents.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Sottilare, R., Brawner, K.: Exploring standardization opportunities by examining interaction between common adaptive instructional system components. In: Proceedings of the First Adaptive Instructional Systems (AIS) Standards Workshop, Orlando, Florida (2018)
Berner, C., et al.: Dota 2 with large scale deep reinforcement learning. ar**v preprint ar**v:1912.06680 (2019)
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (2018)
Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015)
Brockman, G., et al.: OpenAI Gym (2016)
Lake, B.M., Ullman, T.D., Tenenbaum, J.B., Gershman, S.J.: Building machines that learn and think like people. Behav. Brain Sci. 40 (2017)
Taylor, M.E., Stone, P.: Transfer learning for reinforcement learning domains: a survey. J. Mach. Learn. Res. 10 (2009)
van Oijen, J., Roessingh, J.J., Pop**a, G., GarcÃa, V.: Learning analytics of playing space fortress with reinforcement learning. In: International Conference on Human-Computer Interaction. pp. 363–378. Springer (2019). https://doi.org/10.1007/978-3-030-22341-0_29
Oquab, M., Bottou, L., Laptev, I., Sivic, J.: Learning and transferring mid-level image representations using convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1717–1724 (2014)
Brown, T.B., et al.: Language models are few-shot learners. ar**v preprint ar**v:2005.14165 (2020)
M Spronck, P.H., Ponsen, M.J.V., Sprinkhuizen-Kuyper, I.G., Postma, E.O.: Adaptive game AI with dynamic scripting. Mach. Learn. 63, 217–248 (2006)
Kulkarni, T.D., Narasimhan, K., Saeedi, A., Tenenbaum, J.: Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. Adv. Neural. Inf. Process. Syst. 29, 3675–3683 (2016)
Branch, R.M.: Instructional Design: The ADDIE Approach. Springer, Boston (2009). https://doi.org/10.1007/978-0-387-09506-6
Albawi, S., Mohammed, T.A., Al-Zawi, S.: Understanding of a convolutional neural network. In: 2017 International Conference on Engineering and Technology (ICET), pp. 1–6. IEEE (2017)
Bakker, B.: Reinforcement learning with long short-term memory. In: NIPS, pp. 1475–1482 (2001)
Rae, J.W., Potapenko, A., Jayakumar, S.M., Lillicrap, T.P.: Compressive transformers for long-range sequence modelling. ar**v preprint ar**v:1911.05507 (2019)
Hedegaard, M.: The zone of proximal development as basis for instruction. In: Moll, L.C.E. (ed.) Vygotsky and Education: Instructional Implications and Applications of Sociohistorical Psychology, pp. 349–371. Cambridge University Press (1990)
Fleer, S.: Scaffolding for learning from reinforcement: Improving interaction learning (2020)
Laud, A.D.: Theory and application of reward sha** in reinforcement learning (2004)
Niehaus, J., Riedl, M.O.: Scenario adaptation: An approach to customizing computer-based training games and simulations. In: Proceedings of the AIED 2009 Workshop on intelligent Educational Games, pp. 89–98 (2009)
Elman, J.L.: Learning and development in neural networks: the importance of starting small. Cognition 48, 71–99 (1993)
Narvekar, S., Peng, B., Leonetti, M., Sinapov, J., Taylor, M.E., Stone, P.: Curriculum learning for reinforcement learning domains: a framework and survey. J. Mach. Learn. Res. 21, 1–50 (2020)
Hussein, A., Gaber, M.M., Elyan, E., Jayne, C.: Imitation learning: a survey of learning methods. ACM Comput. Surv. (CSUR) 50, 1–35 (2017)
Borsa, D., Piot, B., Munos, R., Pietquin, O.: Observational learning by reinforcement learning. ar**v preprint ar**v:1706.06617 (2017)
Levine, S., Kumar, A., Tucker, G., Fu, J.: Offline reinforcement learning: Tutorial, review, and perspectives on open problems. ar**v preprint ar**v:2005.01643 (2020)
Fu, J., Kumar, A., Nachum, O., Tucker, G., Levine, S.: D4rl: datasets for deep data-driven reinforcement learning. ar**v preprint ar**v:2004.07219 (2020)
Toubman, A.: Validating air combat behaviour models for adaptive training of teams. In: Sottilare, R.A., Schwarz, J. (eds.) HCII 2019. LNCS, vol. 11597, pp. 557–571. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-22341-0_44
Garcıa, J., Fernández, F.: A comprehensive survey on safe reinforcement learning. J. Mach. Learn. Res. 16, 1437–1480 (2015)
Mondesire, S.C., Wiegand, R.P.: A demonstration of stability-plasticity imbalance in multi-agent, decomposition-based learning. In: 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA), pp. 1070–1075. IEEE (2015)
Sottilare, R.: Understanding the AIS problem space. In: Proceedings of the 2nd Adaptive Instructional Systems (AIS) Standards Workshop (2019)
Aubret, A., Matignon, L., Hassas, S.: A survey on intrinsic motivation in reinforcement learning. ar**v preprint ar**v:1908.06976 (2019)
Gupta, A., Eysenbach, B., Finn, C., Levine, S.: Unsupervised meta-learning for reinforcement learning. ar**v preprint ar**v:1806.04640 (2018)
Karli, M., Efe, M.Ö., Sever, H.: Air combat learning from F-16 flight information. In: 2017 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), pp. 1–6. IEEE (2017)
Toubman, A.: Calculated moves: Generating air combat behaviour. Ph.D. dissertation (2020)
Zhang, X., Liu, G., Yang, C., Wu, J.: Research on air confrontation maneuver decision-making method based on reinforcement learning. Electronics 7, 279 (2018)
Doyle, M.J., Portrey, A.M.: Rapid adaptive realistic behavior modeling is viable for use in training. In: Proceedings of the 23rd Conference on Behavior Representation in Modeling and Simulation (BRIMS), pp. 73–80 (2014)
Freeman, J., Watz, E., Bennett, W.: Adaptive agents for adaptive tactical training: the state of the art and emerging requirements. In: Sottilare, R.A., Schwarz, J. (eds.) HCII 2019. LNCS, vol. 11597, pp. 493–504. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-22341-0_39
van Oijen, J., Toubman, A., Pop**a, G.: Effective behaviour modelling for computer generated forces. In: Interservice/Industry Training, Simulation and Education Conference (I/ITSEC). I/ITSEC (2019)
Warwick, W., Rodgers, S.: Wrong in the right way: balancing realism against other constraints in simulation-based training. In: Sottilare, R., Schwarz, J. (eds.) Adaptive Instructional Systems. HCII 2019. LNCS, vol 11597, pp. 379–388. Springer (2019). https://doi.org/10.1007/978-3-030-22341-0_30
Ludwig, J., Presnell, B.: Develo** an adaptive opponent for tactical training. In: Sottilare, R., Schwarz, J. (eds.) Adaptive Instructional Systems. HCII 2019. LNCS, vol 11597. pp. 532–541. Springer (2019). https://doi.org/10.1007/978-3-030-22341-0_42
Luotsinen, L.J., Løvlid, R.A.: Data-driven behavior modeling for computer generated forces. In: NATO Modelling and Simulation Group Symposium M&S Support to Operational Tasks Including War Gaming, Logistics, Cyber Defence (MSG-133), pp. 1–13 (2015)
Sottilare, R.: Exploring methods to promote interoperability in adaptive instructional systems. In: Sottilare, R.A., Schwarz, J. (eds.) HCII 2019. LNCS, vol. 11597, pp. 227–238. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-22341-0_19
Brawner, K.: Bridging conceptual models and architectural interchange for adaptive instructional systems. In: Sottilare, R.A., Schwarz, J. (eds.) HCII 2020. LNCS, vol. 12214, pp. 34–44. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-50788-6_3
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
van Oijen, J., Toubman, A., Claessen, O. (2021). Teaching Reinforcement Learning Agents with Adaptive Instructional Systems. In: Sottilare, R.A., Schwarz, J. (eds) Adaptive Instructional Systems. Design and Evaluation. HCII 2021. Lecture Notes in Computer Science(), vol 12792. Springer, Cham. https://doi.org/10.1007/978-3-030-77857-6_8
Download citation
DOI: https://doi.org/10.1007/978-3-030-77857-6_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-77856-9
Online ISBN: 978-3-030-77857-6
eBook Packages: Computer ScienceComputer Science (R0)