Author Interview - ACCEL: Evolving Curricula with Regret-Based Environment Design

#ai #accel #evolution

This is an interview with the authors Jack Parker-Holder and Minqi Jiang.
Original Paper Review Video: https://www.youtube.com/watch?v=povBD...

Automatic curriculum generation is one of the most promising avenues for Reinforcement Learning today. Multiple approaches have been proposed, each with their own set of advantages and drawbacks. This paper presents ACCEL, which takes the next step into the direction of constructing curricula for multi-capable agents. ACCEL combines the adversarial adaptiveness of regret-based sampling methods with the capabilities of level-editing, usually found in Evolutionary Methods.

OUTLINE:
0:00 - Intro
1:00 - Start of interview
4:45 - How did you get into this field?
8:10 - What is minimax regret?
11:45 - What levels does the regret objective select?
14:20 - Positive value loss (correcting my mistakes)
21:05 - Why is the teacher not learned?
24:45 - How much domain-specific knowledge is needed?
29:30 - What problems is this applicable to?
33:15 - Single agent vs population of agents
37:25 - Measuring and balancing level difficulty
40:35 - How does generalization emerge?
42:50 - Diving deeper into the experimental results
47:00 - What are the unsolved challenges in the field?
50:00 - Where do we go from here?

Website: https://accelagent.github.io
Paper: https://arxiv.org/abs/2203.01302
ICLR Workshop: https://sites.google.com/view/aloe2022
Book on topic: https://www.oreilly.com/radar/open-en...

Abstract:
It remains a significant challenge to train generally capable agents with reinforcement learning (RL). A promising avenue for improving the robustness of RL agents is through the use of curricula. One such class of methods frames environment design as a game between a student and a teacher, using regret-based objectives to produce environment instantiations (or levels) at the frontier of the student agent's capabilities. These methods benefit from their generality, with theoretical guarantees at equilibrium, yet they often struggle to find effective levels in challenging design spaces. By contrast, evolutionary approaches seek to incrementally alter environment complexity, resulting in potentially open-ended learning, but often rely on domain-specific heuristics and vast amounts of computational resources. In this paper we propose to harness the power of evolution in a principled, regret-based curriculum. Our approach, which we call Adversarially Compounding Complexity by Editing Levels (ACCEL), seeks to constantly produce levels at the frontier of an agent's capabilities, resulting in curricula that start simple but become increasingly complex. ACCEL maintains the theoretical benefits of prior regret-based methods, while providing significant empirical gains in a diverse set of environments. An interactive version of the paper is available at this http URL.

Authors: Jack Parker-Holder, Minqi Jiang, Michael Dennis, Mikayel Samvelyan, Jakob Foerster, Edward Grefenstette, Tim Rocktäschel