Let’s begin by giving a proper definition of MDPs:
A Markov Choice Course of is a 5-tuple (S, A, R, P, γ) such that:
- S is the set of states the agent may be in
- A is the set of actions the agent can take
- R : S x A → R the reward perform
- P is the set of likelihood distributions outlined such that P(s’|s,a) is the likelihood of transitioning to state s’ if the agent takes motion a in state s. Notice that MDPs are Markov processes, which means that the Markov property holds on the transition chances: P(Sₜ₊₁|S₀, A₀, …, Sₜ, Aₜ) = P(Sₜ₊₁|Sₜ, Aₜ)
- γ ∈ (0, 1] is a low cost issue. Whereas we often cope with discounted issues (i.e. γ < 1), the formulations offered are additionally legitimate for undiscounted MDPs (γ = 1)
We then outline the coverage, i.e. what dictates the agent’s conduct in an MDP:
A coverage π is a likelihood measure over the motion area outlined as: π(a|s) is the likelihood of taking motion a when the agent is in state s.
We lastly introduce the worth perform, i.e. the agent’s goal in an MDP:
The worth perform of a coverage π is the anticipated discounted reward below this coverage, when beginning at a given state s:
Particularly, the worth perform of the optimum coverage π* satisfies the Bellman optimality equation:
Which yields the deterministic optimum coverage:
Deriving the LP formulation of MDPs:
Given the above definitions, we are able to begin by noticing that any worth perform V that satisfies
is an higher sure on the optimum worth perform. To see it, we are able to begin by noticing that such worth perform additionally satisfies:
We acknowledge the worth iteration operator utilized to V:
i.e.
Additionally noticing that the H*operator is rising, we are able to apply it iteratively to have:
the place we used the property of V* being the fastened level of H*.
Subsequently, discovering V* comes right down to discovering the tightest higher sure V that obeys the above equation, which yields the next formulation:
Right here we added a weight time period akin to the likelihood of beginning in state s. We will see that the above drawback is linear in V and may be rewritten as follows: