Value Iteration

Background

Definition

Pesudo code

Gambler's Problem

Case

Case analysis

Code

Result

Value Iteration

Background

Policy iteration's process is that after value function (policy evaluation) converges, the policy then improved. In fact, there is no need that value function converges because even no the last many sweeps of policy evaluation, the policy can converge to the same result. Therefore, actually we do not need to make policy evaluation converged.

Definition

We can go on several sweeps of policy evaluation, then turn into policy improvement and turn back to the policy evaluation until the policy converges to the optimal policy.

In extreme situations, we can make policy improvement just after one update of one state.

Pesudo code

while judge == True:

judge = False;

for $s\in S$ :

$v_{old}\ =\ v(s)$

$v(s)\ =\ max_a \sum_{s^{'},r^{'}} \ p(s^{'},r^{'}|s,a)[r+\gamma v(s^{'})]$

if $|\ v(s) - v_{old} \ | > \theta$ :

judge = True;

# optimal policy

$\pi(s)\ =\ argmax_a \ sum_{s^{'},a^{'}} \ p(s^{'},a^{'}|s,a)[r+\gamma v(s^{'})]$

Gambler's Problem

Case

Case analysis

S: the money we have : 0-100, v(0)=0,v(100)=1 (constant)

A: the money we stake: 0-min( s,100-s )

R: no imtermediate reward, expect v(100)=1

$S^{'}$ : S+A or S-A, which depends on the probability: $p_h$ (win)

Code

### settingsimport math
import numpy
import random# visualization
import matplotlib
import matplotlib.pyplot as plt# global settingsMAX_MONEY = 100 ;MIN_MONEY = 0 ; P_H = 0.4 ; gamma = 1 ;  # episodic# accuracy of value function
error = 0.001;

### functions# max value function
def max_v_a(v,s):MIN_ACTION = 0 ;MAX_ACTION = min( s, MAX_MONEY-s ) ;v_a = numpy.zeros(MAX_ACTION+1 - MIN_ACTION ,dtype = numpy.float);for a in range(MIN_ACTION, MAX_ACTION+1):v_a [a-MIN_ACTION] = ( P_H*( 0 + gamma*v[s+a] ) + \(1- P_H)*( 0 + gamma*v[s-a] ) );return max(v_a)# max value function index
def argmax_a(v,s):MIN_ACTION = 0 ;MAX_ACTION = min( s, MAX_MONEY-s ) ;v_a = numpy.zeros(MAX_ACTION+1 - MIN_ACTION ,dtype = numpy.float);for a in range(MIN_ACTION, MAX_ACTION+1):v_a [a-MIN_ACTION] = ( P_H*( 0 + gamma*v[s+a] ) + \(1- P_H)*( 0 + gamma*v[s-a] ) );return ( numpy.argmax(v_a) )# visualization def visualization(v_set,policy):fig,axes = plt.subplots(2,1)for i in range(0,len(v_set)):axes[0].plot(v_set[i],linewidth=3)#   plt.pause(0.5)axes[0].set_title('value function')axes[1].plot(range(1,len(policy)+1),policy)axes[1].set_title('policy')plt.show()

### main programming# policy
policy = numpy.zeros(MAX_MONEY-1);# value function
v = numpy.zeros(MAX_MONEY+1,dtype = numpy.float);#every_sweep_of value function
v_set = [] ;# initialization
v[MAX_MONEY] = 1 ;
v[MIN_MONEY] = 0 ;judge = True;# value iterationwhile(judge):judge = Falsefor s in range(MIN_MONEY+1,MAX_MONEY):v_old = v[s];v[s] = max_v_a(v,s);if math.fabs( v[s] - v_old ) > error:judge = True;v_set.append(v.copy())# optimal policy for s in range(MIN_MONEY+1,MAX_MONEY):policy[s-1] = argmax_a(v,s)# visualizationvisualization(v_set,policy)

Result

for $p_h$ =0.4

2.2 DP: Value Iteration Gambler‘s Problem相关推荐

【RL系列】马尔可夫决策过程——Gambler's Problem
Gambler's Problem,即"赌徒问题",是一个经典的动态编程里值迭代应用的问题. 在一个掷硬币游戏中,赌徒先下注,如果硬币为正面,赌徒赢回双倍,若是反面,则输掉赌注.赌 ...
RL(Chapter 4): Gambler’s Problem
本文为强化学习笔记,主要参考以下内容: Reinforcement Learning: An Introduction 代码全部来自 GitHub 习题答案参考 Github 目录 Gambler's ...
【AC自动机+DP】[USACO JAN2012 GOLD Problem 1: Video Game Combos]
题目大意题目大意:给定有ABC组成的串n个,然后请你生成一个长度为K的串求给定的串在生成串中最多被匹配时的次数分析 AC自动机模板题. 构建trie,然后DP. p是当前节点,l是已经构造的串的长 ...
《强化学习》中的时序差分学习 Temporal-Difference Learning （基于与动态规划 DP 、蒙特卡洛方法 MC 的对比）
前言: 学习了 Sutton 的<强化学习(第二版)>中时序差分学习的"预测"部分内容.前两章中,书介绍了动态规划与蒙特卡洛方法 ,我们从二者与时序差分学习的 ...
0x51.动态规划 - 线性DP（习题详解 × 10）
目录 0x51.动态规划 - 线性DP 0x51.1 LIS问题 Problem A. 登山 (最长下降子序列) Problem B. 友好城市(思维) Problem C. 最大上升子序列和 0x5 ...
动态规划 —— 线性 DP
[概述] 线性动态规划,是较常见的一类动态规划问题,其是在线性结构上进行状态转移,这类问题不像背包问题.区间DP等有固定的模板. 线性动态规划的目标函数为特定变量的线性函数,约束是这些变量的线性不等式 ...
题目1452：搬寝室（dp题目）
题目链接:http://ac.jobdu.com/problem.php?pid=1452 详解链接:https://github.com/zpfbuaa/JobduInCPlusPlus 参考代码: ...
刷题之路：DP思想（动态规划）
dp一般用于解决决策问题,比如说你的每一步都有好几种处理方式,怎么选择使得最后的结果满足或者接近于你的预期是需要考虑的问题. 所以dp问题实际上也就是最优解的问题一般采用的方式就是将问题拆分成若干个 ...
动态规划法（四）0-1背包问题（0-1 Knapsack Problem）
继续讲故事~~ 转眼我们的主人公丁丁就要离开自己的家乡,去大城市见世面了.这天晚上,妈妈正在耐心地帮丁丁收拾行李.家里有个最大能承受20kg的袋子,可是妈妈却有很多东西想装袋子里,已知行李的编 ...

2.2 DP: Value Iteration Gambler‘s Problem

Value Iteration

Background

Definition

Pesudo code

Gambler's Problem

Case

Case analysis

Code

Result

2.2 DP: Value Iteration Gambler‘s Problem相关推荐

最新文章

热门文章