Markus Holzleitner, Lukas Gruber, Jose Arjona Medina, Johannes Brandstetter, Sepp Hochreiter,
"Convergence Proof for Actor-Critic Methods Applied to PPO and RUDDER"
: Transactions on Large-Scale Data- and Knowledge-Centered Systems XLVIII. Special Issue In Memory of Univ. Prof. Dr. Roland Wagner, Serie Lecture Notes in Computer Science (LNCS), Vol. 12670, 2021
Original Titel:
Convergence Proof for Actor-Critic Methods Applied to PPO and RUDDER
Sprache des Titels:
Englisch
Original Buchtitel:
Transactions on Large-Scale Data- and Knowledge-Centered Systems XLVIII. Special Issue In Memory of Univ. Prof. Dr. Roland Wagner
Original Kurzfassung:
We prove under commonly used assumptions the convergence of actor-critic reinforcement learning algorithms, which simultaneously learn a policy function, the actor, and a value function, the critic. Both functions can be deep neural networks of arbitrary complexity. Our framework allows showing convergence of the well known Proximal Policy Optimization (PPO) and of the recently introduced RUDDER. For the convergence proof we employ recently introduced techniques from the two time-scale stochastic approximation theory.
Previous convergence proofs assume linear function approximation, cannot treat episodic examples, or do not consider that policies become greedy. The latter is relevant since optimal policies are typically deterministic. Our results are valid for actor-critic methods that use episodic samples and that have a policy that becomes more greedy during learning.
Forschungs-