Pure-Past Action Masking

Giovanni Varricchione; Natasha  Alechina; Mehdi Dastani; Giuseppe  De Giacomo; Brian Logan; Giuseppe  Perelli

doi:10.1609/aaai.v38i19.30163

Pure-Past Action Masking

Giovanni Varricchione, Natasha Alechina, Mehdi Dastani, Giuseppe De Giacomo, Brian Logan, Giuseppe Perelli

Research output: Chapter in Book/Report/Conference proceeding › Published conference contribution

2 Citations (Scopus)

5 Downloads (Pure)

Abstract

We present Pure-Past Action Masking (PPAM), a lightweight approach to action masking for safe reinforcement learning. In PPAM, actions are disallowed (“masked”) according to specifications expressed in Pure-Past Linear Temporal Logic (PPLTL). PPAM can enforce non-Markovian constraints, i.e., constraints based on the history of the system, rather than just the current state of the (possibly hidden) MDP. The features used in the safety constraint need not be the same as those used by the learning agent, allowing a clear separation
of concerns between the safety constraints and reward specifications of the (learning) agent. We prove formally that an agent trained with PPAM can learn any optimal policy that satisfies the safety constraints, and that they are as expressive as shields, another approach to enforce non-Markovian constraints in RL. Finally, we provide empirical results showing how PPAM can guarantee constraint satisfaction in practice.

Original language	English
Title of host publication	AAAI Conference and Symposium Proceedings
Publisher	AAAI Press
Pages	21646-21655
Number of pages	10
Volume	38
Edition	19
ISBN (Electronic)	978-1-57735-887-9
DOIs	https://doi.org/10.1609/aaai.v38i19.30163
Publication status	Published - 25 Mar 2024
Event	The 38th Annual AAAI Conference on Artificial Intelligence - Vancouver Convention Centre, Vancouver, Canada Duration: 20 Feb 2024 → 27 Feb 2024 Conference number: 38 https://aaai.org/aaai-conference/

Publication series

Name	Proceedings of the AAAI Conference on Artificial Intelligence
ISSN (Print)	2159-5399
ISSN (Electronic)	2374-3468

Conference

Conference	The 38th Annual AAAI Conference on Artificial Intelligence
Abbreviated title	AAAI
Country/Territory	Canada
City	Vancouver
Period	20/02/24 → 27/02/24
Internet address	https://aaai.org/aaai-conference/

Funding

This work was supported by PNRR MUR project PE0000013-FAIR, partially supported by ERC Advanced Grant WhiteMech (No. 834228), EU ICT-48 2020 project TAILOR (No. 952215), the ONRG project N62909-22-1- 2005, the InDAM-GNCS project “Strategic Reasoning in Mechanism Design”, and the project OCENW.M.21.377 funded by the Dutch Research Council (NWO). For the purpose of open access, the authors have applied a Creative Commons Attribution (CC BY) licence to any Author Accepted Manuscript version arising from this submission.

Funders	Funder number
Ministero dell'Università e della Ricerca	PE0000013-FAIR
European Research Council	834228
European Commission	952215
Office of Naval Research Global	N62909-22-1- 2005
Gruppo Nazionale per il Calcolo Scientifico
Istituto Nazionale di Alta Matematica
The Dutch Research Council	OCENW.M.21.377

Keywords

General

Access to Document

10.1609/aaai.v38i19.30163Licence: Other

Varricchione_etal_AAAI_Pure_Past_Action_AAM
This manuscript has been made open access under a Creative Commons Attribution (CC BY) licence under the terms of the University of Aberdeen Research Publications Policy. https://creativecommons.org/licenses/by/4.0/
Accepted author manuscript, 431 KBLicence: CC BY

https://aaai.org/aaai-publications/aaai-conference-proceedings/Licence: Unspecified

Cite this

Varricchione, G, Alechina, N, Dastani, M, De Giacomo, G, Logan, B & Perelli, G 2024, Pure-Past Action Masking. in AAAI Conference and Symposium Proceedings. 19 edn, vol. 38, Proceedings of the AAAI Conference on Artificial Intelligence, AAAI Press, pp. 21646-21655, The 38th Annual AAAI Conference on Artificial Intelligence, Vancouver, Canada, 20/02/24. https://doi.org/10.1609/aaai.v38i19.30163

@inproceedings{08a7bed2a1ef4f1791ec44e335984cd0,

title = "Pure-Past Action Masking",

abstract = "We present Pure-Past Action Masking (PPAM), a lightweight approach to action masking for safe reinforcement learning. In PPAM, actions are disallowed (“masked”) according to specifications expressed in Pure-Past Linear Temporal Logic (PPLTL). PPAM can enforce non-Markovian constraints, i.e., constraints based on the history of the system, rather than just the current state of the (possibly hidden) MDP. The features used in the safety constraint need not be the same as those used by the learning agent, allowing a clear separationof concerns between the safety constraints and reward specifications of the (learning) agent. We prove formally that an agent trained with PPAM can learn any optimal policy that satisfies the safety constraints, and that they are as expressive as shields, another approach to enforce non-Markovian constraints in RL. Finally, we provide empirical results showing how PPAM can guarantee constraint satisfaction in practice.",

keywords = "General",

author = "Giovanni Varricchione and Natasha Alechina and Mehdi Dastani and {De Giacomo}, Giuseppe and Brian Logan and Giuseppe Perelli",

year = "2024",

month = mar,

day = "25",

doi = "10.1609/aaai.v38i19.30163",

language = "English",

volume = "38",

series = "Proceedings of the AAAI Conference on Artificial Intelligence",

publisher = "AAAI Press",

pages = "21646--21655",

booktitle = "AAAI Conference and Symposium Proceedings",

edition = "19",

note = "The 38th Annual AAAI Conference on Artificial Intelligence, AAAI ; Conference date: 20-02-2024 Through 27-02-2024",

url = "https://aaai.org/aaai-conference/",

}

TY - GEN

T1 - Pure-Past Action Masking

AU - Varricchione, Giovanni

AU - Alechina, Natasha

AU - Dastani, Mehdi

AU - De Giacomo, Giuseppe

AU - Logan, Brian

AU - Perelli, Giuseppe

N1 - Conference code: 38

PY - 2024/3/25

Y1 - 2024/3/25

N2 - We present Pure-Past Action Masking (PPAM), a lightweight approach to action masking for safe reinforcement learning. In PPAM, actions are disallowed (“masked”) according to specifications expressed in Pure-Past Linear Temporal Logic (PPLTL). PPAM can enforce non-Markovian constraints, i.e., constraints based on the history of the system, rather than just the current state of the (possibly hidden) MDP. The features used in the safety constraint need not be the same as those used by the learning agent, allowing a clear separationof concerns between the safety constraints and reward specifications of the (learning) agent. We prove formally that an agent trained with PPAM can learn any optimal policy that satisfies the safety constraints, and that they are as expressive as shields, another approach to enforce non-Markovian constraints in RL. Finally, we provide empirical results showing how PPAM can guarantee constraint satisfaction in practice.

AB - We present Pure-Past Action Masking (PPAM), a lightweight approach to action masking for safe reinforcement learning. In PPAM, actions are disallowed (“masked”) according to specifications expressed in Pure-Past Linear Temporal Logic (PPLTL). PPAM can enforce non-Markovian constraints, i.e., constraints based on the history of the system, rather than just the current state of the (possibly hidden) MDP. The features used in the safety constraint need not be the same as those used by the learning agent, allowing a clear separationof concerns between the safety constraints and reward specifications of the (learning) agent. We prove formally that an agent trained with PPAM can learn any optimal policy that satisfies the safety constraints, and that they are as expressive as shields, another approach to enforce non-Markovian constraints in RL. Finally, we provide empirical results showing how PPAM can guarantee constraint satisfaction in practice.

KW - General

U2 - 10.1609/aaai.v38i19.30163

DO - 10.1609/aaai.v38i19.30163

M3 - Published conference contribution

VL - 38

T3 - Proceedings of the AAAI Conference on Artificial Intelligence

SP - 21646

EP - 21655

BT - AAAI Conference and Symposium Proceedings

PB - AAAI Press

T2 - The 38th Annual AAAI Conference on Artificial Intelligence

Y2 - 20 February 2024 through 27 February 2024

ER -

Pure-Past Action Masking

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Fingerprint

Cite this