Abstract
As a sequential decision problem, robotic soccer can benefit from research in reinforcement learning. We introduce the 3 vs. 2 keepaway domain, a subproblem of robotic soccer implemented in the RoboCup soccer server. We then explore reinforcement learning methods for policy evaluation and action selection in this distributed, real-time, partially observable, noisy domain. We present empirical results demonstrating that a learned policy can dramatically outperform hand-coded policies.
Chapter PDF
Similar content being viewed by others
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
J. S. Albus. Brains, Behavior, and Robotics. Byte Books, Peterborough, NH, 1981.
T. Andou. Refinement of soccer agents’ positions using reinforcement learning. In H. Kitano, editor, RoboCup-97: Robot Soccer World Cup I, pages 373–388. Springer Verlag, Berlin, 1998.
R. H. Crites and A. G. Barto. Improving elevator performance using reinforcement learning. In D. S. Touretzky, M. C. Mozer, and M. E. Hasselmo, editors, Advances in Neural Processing Systems 8, Cambridge, MA, 1996. MIT Press.
H. Kitano, M. Tambe, P. Stone, M. Veloso, S. Coradeschi, E. Osawa, H. Matsubara, I. Noda, and M. Asada. The RoboCup synthetic agent challenge 97. In Proceedings of the Fifteenth International Joint Conference on Artificial Intelligence, pages 24–29, San Francisco, CA 1997. Morgan Kaufmann.
D. McAllester and P. Stone. Kee** the ball from cmunited-99. In P. Stone, T. Balch, and G. Kraetszchmar, editors, RoboCup-2000: Robot Soccer World Cup IV, Berlin, 2001. Springer Verlag. To appear.
I. Noda, H. Matsubara, K. Hiraki, and I. Frank. Soccer server: A tool for research on multiagent systems. Applied Artificial Intelligence, 12:233-50, 1998.
J. R. Quinlan. C4.5: Programs for Machine Learning. Morgan Kaufmann, San Mateo, CA, 1993.
P. Stone. Layered Learning in Multiagent Systems: A Winning Approach to Robotic Soccer. MIT Press, 2000.
P. Stone, P. Riley, and M. Veloso. The CMUnited-99 champion simulator team. In M. Veloso, E. Pagello, and H. Kitano, editors, RoboCup-99: Robot Soccer World Cup III, pages 35–48. Springer Verlag, Berlin, 2000.
P. Stone and M. Veloso. Team-partitioned, opaque-transition reinforcement learning. In M. Asada and H. Kitano, editors, RoboCup-98: Robot Soccer World Cup II. Springer Verlag, Berlin, 1999. Also in Proceedings of the Third International Conference on Autonomous Agents,1999.
R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. MIT Press, Cambridge,Massachusetts, 1998.
R. S. Sutton and S. D. Whitehead. Online learning with random representations. In Proceedings of the Tenth International Conference on Machine Learning, pages 314-21, 1993.
M. Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceedings of the Tenth International Conference on Machine Learning, pages 330-37, 1993.
E. Uchibe. Cooperative Behavior Acquisition by Learning and Evolution in a Multi-Agent Environment for Mobile Robots. PhD thesis, Osaka University, January 1999.
M. Veloso, P. Stone, and M. Bowling. Anticipation as a key for collaboration in a team of agents: A case study in robotic soccer. In Proceedings of SPIE Sensor Fusion and Decentralized Control in Robotic Systems II, volume 3839, Boston, September 1999.
C. J. C. H. Watkins. Learning from Delayed Rewards. PhD thesis, King’s Cambridge, UK, 1989.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2001 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Stone, P., Sutton, R.S., Singh, S. (2001). Reinforcement Learning for 3 vs. 2 Keepaway. In: Stone, P., Balch, T., Kraetzschmar, G. (eds) RoboCup 2000: Robot Soccer World Cup IV. RoboCup 2000. Lecture Notes in Computer Science(), vol 2019. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-45324-5_23
Download citation
DOI: https://doi.org/10.1007/3-540-45324-5_23
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-42185-6
Online ISBN: 978-3-540-45324-6
eBook Packages: Springer Book Archive