Abstract
We present Pure-Past Action Masking (PPAM), a lightweight approach to action masking for safe reinforcement learning. In PPAM, actions are disallowed (“masked”) according to specifications expressed in Pure-Past Linear Temporal Logic (PPLTL). PPAM can enforce non-Markovian constraints, i.e., constraints based on the history of the system, rather than just the current state of the (possibly hidden) MDP. The features used in the safety constraint need not be the same as those used by the learning agent, allowing a clear separation
of concerns between the safety constraints and reward specifications of the (learning) agent. We prove formally that an agent trained with PPAM can learn any optimal policy that satisfies the safety constraints, and that they are as expressive as shields, another approach to enforce non-Markovian constraints in RL. Finally, we provide empirical results showing how PPAM can guarantee constraint satisfaction in practice.
of concerns between the safety constraints and reward specifications of the (learning) agent. We prove formally that an agent trained with PPAM can learn any optimal policy that satisfies the safety constraints, and that they are as expressive as shields, another approach to enforce non-Markovian constraints in RL. Finally, we provide empirical results showing how PPAM can guarantee constraint satisfaction in practice.
Original language | English |
---|---|
Title of host publication | AAAI Conference and Symposium Proceedings |
Publisher | AAAI Press |
Publication status | Accepted/In press - 18 Feb 2024 |
Event | The 38th Annual AAAI Conference on Artificial Intelligence - Vancouver Convention Centre, Vancouver, Canada Duration: 20 Feb 2024 → 27 Feb 2024 Conference number: 38 https://aaai.org/aaai-conference/ |
Conference
Conference | The 38th Annual AAAI Conference on Artificial Intelligence |
---|---|
Abbreviated title | AAAI |
Country/Territory | Canada |
City | Vancouver |
Period | 20/02/24 → 27/02/24 |
Internet address |