Is such an MDP solvable?