Multi-view reinforcement learning for sequential decision-making with insufficient state information

Abstract

Most reinforcement learning methods describe sequential decision-making as a Markov decision process where the effect of action is only decided by the current state. But this is reasonable only if the state is correctly defined and the state information is sufficiently observed. Thus the learning efficiency of reinforcement learning methods based on Markov decision process is limited when the state information is insufficient. Partially observable Markov decision process and history-based decision process are respectively proposed to describe sequential decision-making with insufficient state information. However, these two processes are easy to ignore the important information from the current observed state. Therefore, the learning efficiency of reinforcement learning methods based on these two processes is also limited when the state information is insufficient. In this paper, we propose a multi-view reinforcement learning method to solve this problem. The motivation is that the interaction information between the agent and its environment should be considered from the views of history, present, and future to overcome the insufficiency of state information. Based on these views, we construct a multi-view decision process to describe sequential decision-making with insufficient state information. A multi-view reinforcement learning method is proposed by combining the multi-view decision process and the actor-critic framework. In the proposed method, multi-view clustering is performed to ensure that each type of sample can be sufficiently exploited. Experiments illustrate that the proposed method is more effective than the compared state-of-the-arts. The source code can be downloaded from https://github.com/jamieliuestc/MVRL.

Appendix

1.1 Proof of Theorem 1

Proof

$$\begin{aligned}{} & {} \Vert B^{\pi }Q_{1}(h,o,a) - B^{\pi }Q_{2}(h,o,a) \Vert _{\infty }\\{} & {} \quad = \max _{h,o,a} \left| \mathbb {E}_{a'\thicksim \pi ,o'\thicksim p^{\pi }}[r(h,o,a) + \gamma Q_{1}(h',o',a') - r(h,o,a) - \gamma Q_{2}(h',o',a')] \right| \\{} & {} \quad = \max _{h,o,a} \left| \sum _{o'\in \mathcal {O}}p(o'|h,o,a)\gamma \sum _{a'\in \mathcal {A}}\pi (a'|h',o') \left[ Q_{1}(h',o',a')-Q_{2}(h',o',a')\right] \right| \\{} & {} \quad \le \max _{h,o,a} \sum _{o'\in \mathcal {O}}p(o'|h,o,a)\gamma \sum _{a'\in \mathcal {A}}\pi (a'|h',o') \left| Q_{1}(h',o',a')-Q_{2}(h',o',a') \right| \\{} & {} \quad \le \max _{h,o,a} \sum _{o'\in \mathcal {O}}p(o'|h,o,a)\gamma \max _{\widetilde{h},\widetilde{o},\widetilde{a}} \left| Q_{1}(\widetilde{h},\widetilde{o},\widetilde{a})-Q_{2}(\widetilde{h},\widetilde{o},\widetilde{a}) \right| \\{} & {} \quad \le \gamma \max _{h,o,a}|Q_{1}(h,o,a)-Q_{2}(h,o,a)|\\{} & {} \quad = \gamma \Vert Q_{1}(h,o,a) - Q_{2}(h,o,a) \Vert _{\infty }. \end{aligned}$$

Thus $B^{\pi }$ (14) is a contraction map** in the sup-norm when MvDP is finite. According to the principle of contraction map**, $B^{\pi }$ has unique fixed point that meets $Q(h,o,a)=\mathbb {E}_{a'\thicksim \pi ,o'\thicksim p^{\pi }}[r(h,o,a)+\gamma Q(h',o',a')]$. According to Bellman equation (13), the unique fixed point is $Q^{\pi }$. $\square$

1.2 Proof of Theorem 2

Proof

$$\begin{aligned}{} & {} \quad \Vert B Q_{1}(h,o,a) - B Q_{2}(h,o,a) \Vert _{\infty }\\{} & {} \quad = \max _{h,o,a} \left| \max _{\pi } \mathbb {E}_{a'\thicksim \pi ,o'\thicksim p^{\pi }}[r(h,o,a) + \gamma Q_{1}(h',o',a')] - \max _{\pi } \mathbb {E}_{a'\thicksim \pi ,o'\thicksim p^{\pi }} [r(h,o,a) + \gamma Q_{2}(h',o',a')] \right| \\{} & {} \quad \le \max _{h,o,a} \left| \max _{\pi } \sum _{o'\in \mathcal {O}}p(o'|h,o,a)\gamma \sum _{a'\in \mathcal {A}}\pi (a'|h',o') \left| Q_{1}(h',o',a')-Q_{2}(h',o',a')\right| \right| \\{} & {} \quad \le \max _{h,o,a,\pi } \sum _{o'\in \mathcal {O}}p(o'|h,o,a)\gamma \sum _{a'\in \mathcal {A}}\pi (a'|h',o') \left| Q_{1}(h',o',a')-Q_{2}(h',o',a') \right| \\{} & {} \quad \le \max _{h,o,a} \sum _{o'\in \mathcal {O}}p(o'|h,o,a)\gamma \max _{\widetilde{h},\widetilde{o},\widetilde{a}} \left| Q_{1}(\widetilde{h},\widetilde{o},\widetilde{a})-Q_{2}(\widetilde{h},\widetilde{o},\widetilde{a}) \right| \\{} & {} \quad \le \gamma \max _{h,o,a}|Q_{1}(h,o,a)-Q_{2}(h,o,a)|\\{} & {} \quad = \gamma \Vert Q_{1}(h,o,a) - Q_{2}(h,o,a) \Vert _{\infty }. \end{aligned}$$

Therefore, B (15) is a contraction map** in the sup-norm when MvDP is finite. According to the principle of contraction map**, B has unique fixed point that meets $Q(h,o,a)=\max _{\pi }\mathbb {E}_{a'\thicksim \pi ,o'\thicksim p^{\pi }}[r(h,o,a)+\gamma Q(h',o',a')]$. According to Bellman equation (13), the unique fixed point is $Q^{\pi ^*}$. $\square$