值函數(shù)可以分為狀態(tài)價(jià)值函數(shù)和動(dòng)作價(jià)值函數(shù)，分別適用于哪些強(qiáng)化學(xué)習(xí)問(wèn)題
二、基于動(dòng)態(tài)規(guī)劃的算法
2.1 策略迭代算法
示例：
(改進(jìn)的) 策略迭代
代碼
首先定義了一些參數(shù)，如獎(jiǎng)勵(lì)、折扣因子、最大誤差等，然后初始化了一個(gè)網(wǎng)格世界的環(huán)境，包括狀態(tài)、動(dòng)作、價(jià)值函數(shù)和策略。接著，它定義了一些函數(shù)，用于打印網(wǎng)格世界的狀態(tài)或策略、計(jì)算狀態(tài)的期望價(jià)值函數(shù)、對(duì)策略進(jìn)行評(píng)估和改進(jìn)等。最后，它使用了策略迭代的算法，不斷地更新策略和價(jià)值函數(shù)，直到找到最優(yōu)的策略，并打印出初始的隨機(jī)策略和最終的最優(yōu)策略。
import?random?#?導(dǎo)入隨機(jī)模塊，用于生成隨機(jī)數(shù)
# Arguments
REWARD = -0.01 # 定義非終止?fàn)顟B(tài)的常數(shù)獎(jiǎng)勵(lì)，這里是負(fù)數(shù)，表示每走一步都有一定的代價(jià)
DISCOUNT = 0.99 # 定義折扣因子，表示未來(lái)獎(jiǎng)勵(lì)的衰減程度，越接近1表示越重視長(zhǎng)期獎(jiǎng)勵(lì)
MAX_ERROR = 10**(-3) # 定義最大誤差，表示當(dāng)狀態(tài)的價(jià)值函數(shù)的變化小于這個(gè)值時(shí)，認(rèn)為收斂到穩(wěn)定值
# Set up the initial environment
NUM_ACTIONS = 4 # 定義動(dòng)作的數(shù)量，這里是4個(gè)，分別是上下左右四個(gè)方向的移動(dòng)。policy里的0~3. # Down, Left, Up, Right
ACTIONS = [(1, 0), (0, -1), (-1, 0), (0, 1)] # 定義動(dòng)作的具體效果，表示每個(gè)動(dòng)作對(duì)行和列的坐標(biāo)的影響，例如(1, 0)表示向下移動(dòng)一格，行坐標(biāo)加1，列坐標(biāo)不變
NUM_ROW = 3 # 定義網(wǎng)格世界的行數(shù)
NUM_COL = 4 # 定義網(wǎng)格世界的列數(shù)
U = [[0, 0, 0, 1], [0, 0, 0, -1], [0, 0, 0, 0], [0, 0, 0, 0]] # 定義每個(gè)狀態(tài)的初始價(jià)值函數(shù)，這里只有兩個(gè)終止?fàn)顟B(tài)有正負(fù)1的獎(jiǎng)勵(lì)，其他狀態(tài)都是0
policy = [[random.randint(0, 3) for j in range(NUM_COL)] for i in range(NUM_ROW)] # 隨機(jī)生成一個(gè)初始策略，每個(gè)狀態(tài)隨機(jī)選擇一個(gè)動(dòng)作，用0到3的整數(shù)表示
# Visualization
def printEnvironment(arr, policy=False): # 定義一個(gè)函數(shù)，用于打印網(wǎng)格世界的狀態(tài)或策略
res = "" # 初始化一個(gè)空字符串，用于存儲(chǔ)打印的結(jié)果
for r in range(NUM_ROW): # 遍歷每一行
res += "|" # 在每一行的開(kāi)頭加上一個(gè)豎線符號(hào)
for c in range(NUM_COL): # 遍歷每一列
if r == c == 1: # 如果是中間的墻壁狀態(tài)，用WALL表示
val = "WALL"
elif r <= 1 and c == 3: # 如果是右上角或右下角的終止?fàn)顟B(tài)，用+1或-1表示
val = "+1" if r == 0 else "-1"
else: # 如果是其他狀態(tài)，用動(dòng)作的方向表示，例如Down, Left, Up, Right
val = ["Down", "Left", "Up", "Right"][arr[r][c]]
res += " " + val[:5].ljust(5) + " |" # 格式化輸出，每個(gè)狀態(tài)占6個(gè)字符，不足的用空格補(bǔ)齊，然后加上一個(gè)豎線符號(hào)
res += "\n" # 每一行結(jié)束后換行
print(res) # 打印結(jié)果
# Get the utility of the state reached by performing the given action from the given state
def getU(U, r, c, action): # 定義一個(gè)函數(shù)，用于獲取從給定的狀態(tài)執(zhí)行給定的動(dòng)作后到達(dá)的狀態(tài)的價(jià)值函數(shù)
dr, dc = ACTIONS[action] # 根據(jù)動(dòng)作的編號(hào)，獲取動(dòng)作對(duì)行和列坐標(biāo)的影響
newR, newC = r+dr, c+dc # 計(jì)算新的行和列坐標(biāo)
if newR < 0 or newC < 0 or newR >= NUM_ROW or newC >= NUM_COL or (newR == newC == 1): # 如果新的坐標(biāo)超出了網(wǎng)格的范圍，或者碰到了墻壁，表示無(wú)效的移動(dòng)
return U[r][c] # 返回原來(lái)的狀態(tài)的價(jià)值函數(shù)
else: # 否則
return U[newR][newC] # 返回新的狀態(tài)的價(jià)值函數(shù)
# Calculate the utility of a state given an action
def calculateU(U, r, c, action): # 定義一個(gè)函數(shù)，用于計(jì)算給定狀態(tài)和動(dòng)作的期望價(jià)值函數(shù)
u = REWARD # 初始化價(jià)值函數(shù)為常數(shù)獎(jiǎng)勵(lì)
????#?狀態(tài)轉(zhuǎn)移概率?0.1左偏?0.8前行?0.1右偏
u += 0.1 * DISCOUNT * getU(U, r, c, (action-1)%4) # 加上向左偏轉(zhuǎn)10%的概率的折扣后的價(jià)值函數(shù)
u += 0.8 * DISCOUNT * getU(U, r, c, action) # 加上按照策略執(zhí)行80%的概率的折扣后的價(jià)值函數(shù)
u += 0.1 * DISCOUNT * getU(U, r, c, (action+1)%4) # 加上向右偏轉(zhuǎn)10%的概率的折扣后的價(jià)值函數(shù)
return u # 返回計(jì)算的結(jié)果
# Perform some simplified value iteration steps to get an approximation of the utilities
def policyEvaluation(policy, U): # 定義一個(gè)函數(shù)，用于對(duì)給定的策略進(jìn)行策略評(píng)估，即計(jì)算策略對(duì)應(yīng)的價(jià)值函數(shù)
while True: # 不斷循環(huán)，直到收斂
nextU = [[0, 0, 0, 1], [0, 0, 0, -1], [0, 0, 0, 0], [0, 0, 0, 0]] # 初始化下一輪的價(jià)值函數(shù)，終止?fàn)顟B(tài)的價(jià)值函數(shù)不變，其他狀態(tài)為0
????????error?=?0?#?初始化最大誤差為0。?相鄰兩次迭代的價(jià)值函數(shù)的差值
for r in range(NUM_ROW): # 遍歷每一行
for c in range(NUM_COL): # 遍歷每一列
if (r <= 1 and c == 3) or (r == c == 1): # 如果是終止?fàn)顟B(tài)或墻壁狀態(tài)，跳過(guò)
continue
????????????????nextU[r][c]?=?calculateU(U,?r,?c,?policy[r][c])#?更新?tīng)顟B(tài)價(jià)值函數(shù)?#?根據(jù)當(dāng)前的價(jià)值函數(shù)和策略，計(jì)算下一輪的價(jià)值函數(shù)，這里使用了簡(jiǎn)化的貝爾曼更新公式
????????????????error?=?max(error,?abs(nextU[r][c]-U[r][c]))?#?更新?tīng)顟B(tài)價(jià)值函數(shù)最大差值
U = nextU # 用下一輪的價(jià)值函數(shù)替換當(dāng)前的價(jià)值函數(shù)
????????if?error?<?MAX_ERROR?*?(1-DISCOUNT)?/?DISCOUNT:?#?如果最大誤差小于一個(gè)閾值（MAX_ERROR?*?(1-DISCOUNT)?/?DISCOUNT??人工設(shè)定的），表示收斂
break # 退出循環(huán)
return U # 返回收斂后的價(jià)值函數(shù)
def policyIteration(policy, U): # 定義一個(gè)函數(shù)，用于對(duì)給定的策略進(jìn)行策略迭代，即不斷地進(jìn)行策略評(píng)估和策略改進(jìn)，直到找到最優(yōu)策略
print("During the policy iteration:\n") # 打印提示信息
while True: # 不斷循環(huán)，直到收斂
U = policyEvaluation(policy, U) # 對(duì)當(dāng)前的策略進(jìn)行策略評(píng)估，得到價(jià)值函數(shù)
unchanged = True # 初始化一個(gè)標(biāo)志，表示策略是否發(fā)生了改變
for r in range(NUM_ROW): # 遍歷每一行
for c in range(NUM_COL): # 遍歷每一列
if (r <= 1 and c == 3) or (r == c == 1): # 如果是終止?fàn)顟B(tài)或墻壁狀態(tài)，跳過(guò)
continue
maxAction, maxU = None, -float("inf") # 初始化最大價(jià)值函數(shù)和對(duì)應(yīng)的動(dòng)作為無(wú)窮小和空值
for action in range(NUM_ACTIONS): # 遍歷所有可能的動(dòng)作
u = calculateU(U, r, c, action) # 計(jì)算當(dāng)前狀態(tài)和動(dòng)作的期望價(jià)值函數(shù)
if u > maxU: # 如果當(dāng)前的價(jià)值函數(shù)大于最大價(jià)值函數(shù)
maxAction, maxU = action, u # 更新最大價(jià)值函數(shù)和對(duì)應(yīng)的動(dòng)作
if maxU > calculateU(U, r, c, policy[r][c]): # 如果最大價(jià)值函數(shù)大于當(dāng)前策略（所確定動(dòng)作）對(duì)應(yīng)的價(jià)值函數(shù)
policy[r][c] = maxAction # 更新策略為最大價(jià)值函數(shù)對(duì)應(yīng)的動(dòng)作
unchanged = False # 標(biāo)記策略發(fā)生了改變
if unchanged: # 如果策略沒(méi)有發(fā)生改變，表示已經(jīng)找到了最優(yōu)策略
break # 退出循環(huán)
printEnvironment(policy) # 打印當(dāng)前的策略
return policy # 返回最優(yōu)策略
# Print the initial environment
print("The initial random policy is:\n") # 打印提示信息
printEnvironment(policy) # 打印初始的隨機(jī)策略
# Policy iteration
policy = policyIteration(policy, U) # 調(diào)用策略迭代函數(shù)，得到最優(yōu)策略
# Print the optimal policy
print("The optimal policy is:\n") # 打印提示信息
printEnvironment(policy) # 打印最優(yōu)策略?輸出結(jié)果：
The initial random policy is:
| Up | Right | Right | +1 |
| Down | WALL | Left | -1 |
| Left | Up | Right | Left |
During the policy iteration:
| Up | Right | Right | +1 |
| Up | WALL | Up | -1 |
| Right | Right | Up | Down |
| Right | Right | Right | +1 |
| Up | WALL | Left | -1 |
| Up | Right | Up | Down |
| Right | Right | Right | +1 |
| Up | WALL | Left | -1 |
| Up | Left | Up | Down |
| Right | Right | Right | +1 |
| Up | WALL | Left | -1 |
| Up | Left | Left | Down |
The optimal policy is:
| Right | Right | Right | +1 |
| Up | WALL | Left | -1 |
| Up | Left | Left | Down |2.2? 價(jià)值迭代算法
?示例-價(jià)值迭代算法
代碼
首先定義了一些參數(shù)，如獎(jiǎng)勵(lì)、折扣因子、最大誤差等，然后初始化了一個(gè)網(wǎng)格世界的環(huán)境，包括狀態(tài)、動(dòng)作、價(jià)值函數(shù)。接著，它定義了一些函數(shù)，用于打印網(wǎng)格世界的狀態(tài)或策略、計(jì)算狀態(tài)的期望價(jià)值函數(shù)、對(duì)價(jià)值函數(shù)進(jìn)行價(jià)值迭代等。最后，它使用了價(jià)值迭代的算法，不斷地更新價(jià)值函數(shù)，直到收斂，并根據(jù)價(jià)值函數(shù)獲取最優(yōu)的策略，并打印出初始的價(jià)值函數(shù)和最終的最優(yōu)策略。
REWARD = -0.01 # 定義非終止?fàn)顟B(tài)的常數(shù)獎(jiǎng)勵(lì)，這里是負(fù)數(shù)，表示每走一步都有一定的代價(jià)
DISCOUNT = 0.99 # 定義折扣因子，表示未來(lái)獎(jiǎng)勵(lì)的衰減程度，越接近1表示越重視長(zhǎng)期獎(jiǎng)勵(lì)
MAX_ERROR = 10**(-3) # 定義最大誤差，表示當(dāng)狀態(tài)的價(jià)值函數(shù)的變化小于這個(gè)值時(shí)，認(rèn)為收斂到穩(wěn)定值
# Set up the initial environment
NUM_ACTIONS = 4 # 定義動(dòng)作的數(shù)量，這里是4個(gè)，分別是上下左右四個(gè)方向的移動(dòng)
ACTIONS = [(1, 0), (0, -1), (-1, 0), (0, 1)] # 定義動(dòng)作的具體效果，表示每個(gè)動(dòng)作對(duì)行和列的坐標(biāo)的影響，例如(1, 0)表示向下移動(dòng)一格，行坐標(biāo)加1，列坐標(biāo)不變
NUM_ROW = 3 # 定義網(wǎng)格世界的行數(shù)
NUM_COL = 4 # 定義網(wǎng)格世界的列數(shù)
U = [[0, 0, 0, 1], [0, 0, 0, -1], [0, 0, 0, 0], [0, 0, 0, 0]] # 定義每個(gè)狀態(tài)的初始價(jià)值函數(shù)，這里只有兩個(gè)終止?fàn)顟B(tài)有正負(fù)1的獎(jiǎng)勵(lì)，其他狀態(tài)都是0
# Visualization
def printEnvironment(arr, policy=False): # 定義一個(gè)函數(shù)，用于打印網(wǎng)格世界的狀態(tài)或策略
res = "" # 初始化一個(gè)空字符串，用于存儲(chǔ)打印的結(jié)果
for r in range(NUM_ROW): # 遍歷每一行
res += "|" # 在每一行的開(kāi)頭加上一個(gè)豎線符號(hào)
for c in range(NUM_COL): # 遍歷每一列
if r == c == 1: # 如果是中間的墻壁狀態(tài)，用WALL表示
val = "WALL"
elif r <= 1 and c == 3: # 如果是右上角或右下角的終止?fàn)顟B(tài)，用+1或-1表示
val = "+1" if r == 0 else "-1"
else: # 如果是其他狀態(tài)
if policy: # 如果是打印策略，用動(dòng)作的方向表示，例如Down, Left, Up, Right
val = ["Down", "Left", "Up", "Right"][arr[r][c]]
else: # 如果是打印價(jià)值函數(shù)，用狀態(tài)的價(jià)值表示，轉(zhuǎn)換為字符串
val = str(arr[r][c])
res += " " + val[:5].ljust(5) + " |" # 格式化輸出，每個(gè)狀態(tài)占6個(gè)字符，不足的用空格補(bǔ)齊，然后加上一個(gè)豎線符號(hào)
res += "\n" # 每一行結(jié)束后換行
print(res) # 打印結(jié)果
# Get the utility of the state reached by performing the given action from the given state
def getU(U, r, c, action): # 定義一個(gè)函數(shù)，用于獲取從給定的狀態(tài)執(zhí)行給定的動(dòng)作后到達(dá)的狀態(tài)的價(jià)值函數(shù)
dr, dc = ACTIONS[action] # 根據(jù)動(dòng)作的編號(hào)，獲取動(dòng)作對(duì)行和列坐標(biāo)的影響
newR, newC = r+dr, c+dc # 計(jì)算新的行和列坐標(biāo)
if newR < 0 or newC < 0 or newR >= NUM_ROW or newC >= NUM_COL or (newR == newC == 1): # 如果新的坐標(biāo)超出了網(wǎng)格的范圍，或者碰到了墻壁，表示無(wú)效的移動(dòng)
return U[r][c] # 返回原來(lái)的狀態(tài)的價(jià)值函數(shù)
else: # 否則
return U[newR][newC] # 返回新的狀態(tài)的價(jià)值函數(shù)
# Calculate the utility of a state given an action
def calculateU(U, r, c, action): # 定義一個(gè)函數(shù)，用于計(jì)算給定狀態(tài)和動(dòng)作的期望價(jià)值函數(shù)
u = REWARD # 初始化價(jià)值函數(shù)為常數(shù)獎(jiǎng)勵(lì)
u += 0.1 * DISCOUNT * getU(U, r, c, (action-1)%4) # 加上向左偏轉(zhuǎn)10%的概率的折扣后的價(jià)值函數(shù)
u += 0.8 * DISCOUNT * getU(U, r, c, action) # 加上按照策略執(zhí)行80%的概率的折扣后的價(jià)值函數(shù)
u += 0.1 * DISCOUNT * getU(U, r, c, (action+1)%4) # 加上向右偏轉(zhuǎn)10%的概率的折扣后的價(jià)值函數(shù)
return u # 返回計(jì)算的結(jié)果
#價(jià)值迭代利用貝爾曼最優(yōu)性方程來(lái)鞥新?tīng)顟B(tài)價(jià)值函數(shù)，每次選擇在當(dāng)前狀態(tài)下達(dá)到最優(yōu)值的動(dòng)作
def valueIteration(U): # 定義一個(gè)函數(shù)，用于對(duì)價(jià)值函數(shù)進(jìn)行價(jià)值迭代，即不斷地更新價(jià)值函數(shù)，直到收斂
print("During the value iteration:\n") # 打印提示信息
while True: # 不斷循環(huán)，直到收斂
nextU = [[0, 0, 0, 1], [0, 0, 0, -1], [0, 0, 0, 0], [0, 0, 0, 0]] # 初始化下一輪的價(jià)值函數(shù)，終止?fàn)顟B(tài)的價(jià)值函數(shù)不變，其他狀態(tài)為0
error = 0 # 初始化最大誤差為0
for r in range(NUM_ROW): # 遍歷每一行
for c in range(NUM_COL): # 遍歷每一列
if (r <= 1 and c == 3) or (r == c == 1): # 如果是終止?fàn)顟B(tài)或墻壁狀態(tài)，跳過(guò)
continue
nextU[r][c] = max([calculateU(U, r, c, action) for action in range(NUM_ACTIONS)]) # 根據(jù)當(dāng)前的價(jià)值函數(shù)和所有可能的動(dòng)作，計(jì)算下一輪的價(jià)值函數(shù)，這里使用了貝爾曼最優(yōu)化更新公式
error = max(error, abs(nextU[r][c]-U[r][c])) # 更新最大誤差
U = nextU # 用下一輪的價(jià)值函數(shù)替換當(dāng)前的價(jià)值函數(shù)
printEnvironment(U) # 打印當(dāng)前的價(jià)值函數(shù)
if error < MAX_ERROR * (1-DISCOUNT) / DISCOUNT: # 如果最大誤差小于一個(gè)閾值，表示收斂
break # 退出循環(huán)
return U # 返回收斂后的價(jià)值函數(shù)
# Get the optimal policy from U
def getOptimalPolicy(U): # 定義一個(gè)函數(shù)，用于根據(jù)價(jià)值函數(shù)獲取最優(yōu)策略，即每個(gè)狀態(tài)選擇使得價(jià)值函數(shù)最大的動(dòng)作
policy = [[-1, -1, -1, -1] for i in range(NUM_ROW)] # 初始化策略為-1，表示無(wú)效的動(dòng)作
for r in range(NUM_ROW): # 遍歷每一行
for c in range(NUM_COL): # 遍歷每一列
if (r <= 1 and c == 3) or (r == c == 1): # 如果是終止?fàn)顟B(tài)或墻壁狀態(tài)，跳過(guò)
continue
# Choose the action that maximizes the utility
maxAction, maxU = None, -float("inf") # 初始化最大價(jià)值函數(shù)和對(duì)應(yīng)的動(dòng)作為無(wú)窮小和空值
for action in range(NUM_ACTIONS): # 遍歷所有可能的動(dòng)作
u = calculateU(U, r, c, action) # 計(jì)算當(dāng)前狀態(tài)和動(dòng)作的期望價(jià)值函數(shù)
if u > maxU: # 如果當(dāng)前的價(jià)值函數(shù)大于最大價(jià)值函數(shù)
maxAction, maxU = action, u # 更新最大價(jià)值函數(shù)和對(duì)應(yīng)的動(dòng)作
policy[r][c] = maxAction # 將當(dāng)前狀態(tài)的策略更新為最大價(jià)值函數(shù)對(duì)應(yīng)的動(dòng)作
return policy # 返回最優(yōu)策略
# Print the initial environment
print("The initial U is:\n") # 打印提示信息
printEnvironment(U) # 打印初始的價(jià)值函數(shù)
# Value iteration
U = valueIteration(U) # 調(diào)用價(jià)值迭代函數(shù)，得到收斂后的價(jià)值函數(shù)
# Get the optimal policy from U and print it
policy = getOptimalPolicy(U) # 調(diào)用獲取最優(yōu)策略的函數(shù)，得到最優(yōu)策略
print("The optimal policy is:\n") # 打印提示信息
printEnvironment(policy,?True)?#?打印最優(yōu)策略輸出結(jié)果：
The initial U is:
| 0 | 0 | 0 | +1 |
| 0 | WALL | 0 | -1 |
| 0 | 0 | 0 | 0 |
During the value iteration:
| -0.01 | -0.01 | 0.782 | +1 |
| -0.01 | WALL | -0.01 | -1 |
| -0.01 | -0.01 | -0.01 | -0.01 |
| -0.01 | 0.607 | 0.858 | +1 |
| -0.01 | WALL | 0.509 | -1 |
| -0.01 | -0.01 | -0.01 | -0.01 |
| 0.467 | 0.790 | 0.917 | +1 |
| -0.02 | WALL | 0.621 | -1 |
| -0.02 | -0.02 | 0.389 | -0.02 |
| 0.659 | 0.873 | 0.934 | +1 |
| 0.354 | WALL | 0.679 | -1 |
| -0.03 | 0.292 | 0.476 | 0.196 |
| 0.781 | 0.902 | 0.941 | +1 |
| 0.582 | WALL | 0.698 | -1 |
| 0.295 | 0.425 | 0.576 | 0.287 |
| 0.840 | 0.914 | 0.944 | +1 |
| 0.724 | WALL | 0.705 | -1 |
| 0.522 | 0.530 | 0.613 | 0.375 |
| 0.869 | 0.919 | 0.945 | +1 |
| 0.798 | WALL | 0.708 | -1 |
| 0.667 | 0.580 | 0.638 | 0.414 |
| 0.883 | 0.920 | 0.945 | +1 |
| 0.836 | WALL | 0.709 | -1 |
| 0.746 | 0.634 | 0.649 | 0.437 |
| 0.889 | 0.921 | 0.945 | +1 |
| 0.854 | WALL | 0.710 | -1 |
| 0.789 | 0.706 | 0.658 | 0.449 |
| 0.892 | 0.921 | 0.945 | +1 |
| 0.863 | WALL | 0.711 | -1 |
| 0.815 | 0.754 | 0.685 | 0.456 |
| 0.893 | 0.921 | 0.946 | +1 |
| 0.867 | WALL | 0.714 | -1 |
| 0.829 | 0.785 | 0.726 | 0.478 |
| 0.894 | 0.921 | 0.946 | +1 |
| 0.869 | WALL | 0.721 | -1 |
| 0.837 | 0.802 | 0.754 | 0.513 |
| 0.894 | 0.922 | 0.947 | +1 |
| 0.870 | WALL | 0.730 | -1 |
| 0.841 | 0.811 | 0.771 | 0.539 |
| 0.895 | 0.922 | 0.948 | +1 |
| 0.870 | WALL | 0.738 | -1 |
| 0.843 | 0.816 | 0.781 | 0.555 |
| 0.895 | 0.923 | 0.948 | +1 |
| 0.871 | WALL | 0.746 | -1 |
| 0.844 | 0.819 | 0.787 | 0.565 |
| 0.896 | 0.924 | 0.949 | +1 |
| 0.871 | WALL | 0.752 | -1 |
| 0.844 | 0.820 | 0.790 | 0.571 |
| 0.897 | 0.925 | 0.950 | +1 |
| 0.872 | WALL | 0.758 | -1 |
| 0.845 | 0.821 | 0.792 | 0.577 |
| 0.898 | 0.926 | 0.951 | +1 |
| 0.873 | WALL | 0.763 | -1 |
| 0.846 | 0.822 | 0.794 | 0.583 |
| 0.898 | 0.926 | 0.951 | +1 |
| 0.874 | WALL | 0.767 | -1 |
| 0.846 | 0.822 | 0.795 | 0.588 |
| 0.899 | 0.927 | 0.952 | +1 |
| 0.874 | WALL | 0.770 | -1 |
| 0.847 | 0.823 | 0.796 | 0.592 |
| 0.900 | 0.927 | 0.952 | +1 |
| 0.875 | WALL | 0.773 | -1 |
| 0.848 | 0.824 | 0.797 | 0.596 |
| 0.900 | 0.928 | 0.952 | +1 |
| 0.876 | WALL | 0.775 | -1 |
| 0.849 | 0.825 | 0.798 | 0.600 |
| 0.900 | 0.928 | 0.953 | +1 |
| 0.876 | WALL | 0.777 | -1 |
| 0.849 | 0.825 | 0.799 | 0.604 |
| 0.901 | 0.928 | 0.953 | +1 |
| 0.877 | WALL | 0.779 | -1 |
| 0.850 | 0.826 | 0.800 | 0.607 |
| 0.901 | 0.928 | 0.953 | +1 |
| 0.877 | WALL | 0.781 | -1 |
| 0.850 | 0.827 | 0.800 | 0.610 |
| 0.901 | 0.929 | 0.953 | +1 |
| 0.877 | WALL | 0.782 | -1 |
| 0.851 | 0.827 | 0.801 | 0.613 |
| 0.902 | 0.929 | 0.953 | +1 |
| 0.878 | WALL | 0.783 | -1 |
| 0.851 | 0.827 | 0.802 | 0.615 |
| 0.902 | 0.929 | 0.953 | +1 |
| 0.878 | WALL | 0.784 | -1 |
| 0.851 | 0.828 | 0.802 | 0.618 |
| 0.902 | 0.929 | 0.954 | +1 |
| 0.878 | WALL | 0.785 | -1 |
| 0.851 | 0.828 | 0.803 | 0.620 |
| 0.902 | 0.929 | 0.954 | +1 |
| 0.878 | WALL | 0.785 | -1 |
| 0.852 | 0.828 | 0.803 | 0.622 |
| 0.902 | 0.929 | 0.954 | +1 |
| 0.878 | WALL | 0.786 | -1 |
| 0.852 | 0.828 | 0.803 | 0.623 |
| 0.902 | 0.929 | 0.954 | +1 |
| 0.878 | WALL | 0.786 | -1 |
| 0.852 | 0.829 | 0.803 | 0.625 |
| 0.902 | 0.929 | 0.954 | +1 |
| 0.879 | WALL | 0.787 | -1 |
| 0.852 | 0.829 | 0.804 | 0.626 |
| 0.902 | 0.929 | 0.954 | +1 |
| 0.879 | WALL | 0.787 | -1 |
| 0.852 | 0.829 | 0.804 | 0.628 |
| 0.902 | 0.930 | 0.954 | +1 |
| 0.879 | WALL | 0.787 | -1 |
| 0.852 | 0.829 | 0.804 | 0.629 |
| 0.903 | 0.930 | 0.954 | +1 |
| 0.879 | WALL | 0.788 | -1 |
| 0.852 | 0.829 | 0.804 | 0.630 |
| 0.903 | 0.930 | 0.954 | +1 |
| 0.879 | WALL | 0.788 | -1 |
| 0.852 | 0.829 | 0.804 | 0.631 |
| 0.903 | 0.930 | 0.954 | +1 |
| 0.879 | WALL | 0.788 | -1 |
| 0.852 | 0.829 | 0.804 | 0.632 |
| 0.903 | 0.930 | 0.954 | +1 |
| 0.879 | WALL | 0.788 | -1 |
| 0.853 | 0.829 | 0.804 | 0.632 |
| 0.903 | 0.930 | 0.954 | +1 |
| 0.879 | WALL | 0.788 | -1 |
| 0.853 | 0.829 | 0.805 | 0.633 |
| 0.903 | 0.930 | 0.954 | +1 |
| 0.879 | WALL | 0.788 | -1 |
| 0.853 | 0.829 | 0.805 | 0.634 |
| 0.903 | 0.930 | 0.954 | +1 |
| 0.879 | WALL | 0.789 | -1 |
| 0.853 | 0.829 | 0.805 | 0.634 |
| 0.903 | 0.930 | 0.954 | +1 |
| 0.879 | WALL | 0.789 | -1 |
| 0.853 | 0.830 | 0.805 | 0.635 |
| 0.903 | 0.930 | 0.954 | +1 |
| 0.879 | WALL | 0.789 | -1 |
| 0.853 | 0.830 | 0.805 | 0.635 |
| 0.903 | 0.930 | 0.954 | +1 |
| 0.879 | WALL | 0.789 | -1 |
| 0.853 | 0.830 | 0.805 | 0.636 |
| 0.903 | 0.930 | 0.954 | +1 |
| 0.879 | WALL | 0.789 | -1 |
| 0.853 | 0.830 | 0.805 | 0.636 |
| 0.903 | 0.930 | 0.954 | +1 |
| 0.879 | WALL | 0.789 | -1 |
| 0.853 | 0.830 | 0.805 | 0.636 |
| 0.903 | 0.930 | 0.954 | +1 |
| 0.879 | WALL | 0.789 | -1 |
| 0.853 | 0.830 | 0.805 | 0.637 |
| 0.903 | 0.930 | 0.954 | +1 |
| 0.879 | WALL | 0.789 | -1 |
| 0.853 | 0.830 | 0.805 | 0.637 |
| 0.903 | 0.930 | 0.954 | +1 |
| 0.879 | WALL | 0.789 | -1 |
| 0.853 | 0.830 | 0.805 | 0.637 |
| 0.903 | 0.930 | 0.954 | +1 |
| 0.879 | WALL | 0.789 | -1 |
| 0.853 | 0.830 | 0.805 | 0.637 |
| 0.903 | 0.930 | 0.954 | +1 |
| 0.879 | WALL | 0.789 | -1 |
| 0.853 | 0.830 | 0.805 | 0.638 |
| 0.903 | 0.930 | 0.954 | +1 |
| 0.879 | WALL | 0.789 | -1 |
| 0.853 | 0.830 | 0.805 | 0.638 |
| 0.903 | 0.930 | 0.954 | +1 |
| 0.879 | WALL | 0.789 | -1 |
| 0.853 | 0.830 | 0.805 | 0.638 |
| 0.903 | 0.930 | 0.954 | +1 |
| 0.879 | WALL | 0.789 | -1 |
| 0.853 | 0.830 | 0.805 | 0.638 |
| 0.903 | 0.930 | 0.954 | +1 |
| 0.879 | WALL | 0.789 | -1 |
| 0.853 | 0.830 | 0.805 | 0.638 |
| 0.903 | 0.930 | 0.954 | +1 |
| 0.879 | WALL | 0.789 | -1 |
| 0.853 | 0.830 | 0.805 | 0.638 |
| 0.903 | 0.930 | 0.954 | +1 |
| 0.879 | WALL | 0.789 | -1 |
| 0.853 | 0.830 | 0.805 | 0.638 |
| 0.903 | 0.930 | 0.954 | +1 |
| 0.879 | WALL | 0.789 | -1 |
| 0.853 | 0.830 | 0.805 | 0.639 |
| 0.903 | 0.930 | 0.954 | +1 |
| 0.879 | WALL | 0.789 | -1 |
| 0.853 | 0.830 | 0.805 | 0.639 |
| 0.903 | 0.930 | 0.954 | +1 |
| 0.879 | WALL | 0.789 | -1 |
| 0.853 | 0.830 | 0.805 | 0.639 |
| 0.903 | 0.930 | 0.954 | +1 |
| 0.879 | WALL | 0.789 | -1 |
| 0.853 | 0.830 | 0.805 | 0.639 |
| 0.903 | 0.930 | 0.954 | +1 |
| 0.879 | WALL | 0.789 | -1 |
| 0.853 | 0.830 | 0.805 | 0.639 |
| 0.903 | 0.930 | 0.954 | +1 |
| 0.879 | WALL | 0.789 | -1 |
| 0.853 | 0.830 | 0.805 | 0.639 |
| 0.903 | 0.930 | 0.954 | +1 |
| 0.879 | WALL | 0.789 | -1 |
| 0.853 | 0.830 | 0.805 | 0.639 |
| 0.903 | 0.930 | 0.954 | +1 |
| 0.879 | WALL | 0.789 | -1 |
| 0.853 | 0.830 | 0.805 | 0.639 |
| 0.903 | 0.930 | 0.954 | +1 |
| 0.879 | WALL | 0.789 | -1 |
| 0.853 | 0.830 | 0.805 | 0.639 |
| 0.903 | 0.930 | 0.954 | +1 |
| 0.879 | WALL | 0.789 | -1 |
| 0.853 | 0.830 | 0.805 | 0.639 |
| 0.903 | 0.930 | 0.954 | +1 |
| 0.879 | WALL | 0.789 | -1 |
| 0.853 | 0.830 | 0.805 | 0.639 |
| 0.903 | 0.930 | 0.954 | +1 |
| 0.879 | WALL | 0.789 | -1 |
| 0.853 | 0.830 | 0.805 | 0.639 |
| 0.903 | 0.930 | 0.954 | +1 |
| 0.879 | WALL | 0.789 | -1 |
| 0.853 | 0.830 | 0.805 | 0.639 |
| 0.903 | 0.930 | 0.954 | +1 |
| 0.879 | WALL | 0.789 | -1 |
| 0.853 | 0.830 | 0.805 | 0.639 |
| 0.903 | 0.930 | 0.954 | +1 |
| 0.879 | WALL | 0.789 | -1 |
| 0.903 | 0.930 | 0.954 | +1 |
| 0.879 | WALL | 0.789 | -1 |
| 0.853 | 0.830 | 0.805 | 0.639 |
| 0.903 | 0.930 | 0.954 | +1 |
| 0.879 | WALL | 0.789 | -1 |
| 0.853 | 0.830 | 0.805 | 0.639 |
The optimal policy is:
| Right | Right | Right | +1 |
| Up | WALL | Left | -1 |
| Up | Left | Left | Down |2.3?價(jià)值迭代算法與策略迭代算法對(duì)比
策略迭代算法和價(jià)值迭代算法都依賴于環(huán)境的模型，需要知道狀態(tài)轉(zhuǎn)移概率和回報(bào)函數(shù)，因此被稱為有模型的強(qiáng)化學(xué)習(xí)算法。
文章來(lái)源：http://www.zghlxwxcb.cn/news/detail-811372.html
The End文章來(lái)源地址http://www.zghlxwxcb.cn/news/detail-811372.html
到了這里，關(guān)于【機(jī)器學(xué)習(xí)】強(qiáng)化學(xué)習(xí)（二）基于動(dòng)態(tài)規(guī)劃的算法的文章就介紹完了。如果您還想了解更多內(nèi)容，請(qǐng)?jiān)谟疑辖撬阉鱐OY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！
