Markov Decision Process

A markov decision process, MDP for short, is defined by the tuple $(\mathcal{S},\mathcal{A},{\mathcal{P}}_a,{\mathcal{R}}_a)$ where

• $\mathcal{S}$ is the set of all possible states, also called the state space
• $\mathcal{A}$ is the set of all possible actions, also known as the action space
• ${\mathcal{P}}_a$ is a probability function on $\mathcal{S}\times \mathcal{S}$ such that ${\mathcal{P}}_a(s,s^{\prime })$ is the probability of endig up in state $s^{\prime }$ upon taking action $a$ at state $s$. (todo, this could be explained in more detail)
• ${\mathcal{R}}_a:\mathcal{S}\times \mathcal{S}\to \mathbb{R}\forall a\in \mathcal{A}$ such that ${\mathcal{R}}_a(s,s^{\prime })$ represents reward recieved after taking action $a$ at state $s$ that lead to gettting to state $s^{\prime }$.

Note, that in a markov decision process the next state is only reliant the previous state, all the other states that have come before are not taken into count.