Provable Defense against Backdoor Policies in Reinforcement Learning (original) (raw)
Abstract:We propose a provable defense mechanism against backdoor policies in reinforcement learning under subspace trigger assumption. A backdoor policy is a security threat where an adversary publishes a seemingly well-behaved policy which in fact allows hidden triggers. During deployment, the adversary can modify observed states in a particular way to trigger unexpected actions and harm the agent. We assume the agent does not have the resources to re-train a good policy. Instead, our defense mechanism sanitizes the backdoor policy by projecting observed states to a 'safe subspace', estimated from a small number of interactions with a clean (non-triggered) environment. Our sanitized policy achieves epsilon\epsilonepsilon approximate optimality in the presence of triggers, provided the number of clean interactions is Oleft(fracD(1−gamma)4epsilon2right)O\left(\frac{D}{(1-\gamma)^4 \epsilon^2}\right)Oleft(fracD(1−gamma)4epsilon2right) where gamma\gammagamma is the discounting factor and DDD is the dimension of state space. Empirically, we show that our sanitization defense performs well on two Atari game environments.
Submission history
From: Shubham Kumar Bharti [view email]
[v1] Fri, 18 Nov 2022 23:12:24 UTC (279 KB)