A Game Theoretical Framework for Adversarial Learning (original) (raw)
Related papers
Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, 2004
Essentially all data mining algorithms assume that the datagenerating process is independent of the data miner's activities. However, in many domains, including spam detection, intrusion detection, fraud detection, surveillance and counter-terrorism, this is far from the case: the data is actively manipulated by an adversary seeking to make the classifier produce false negatives. In these domains, the performance of a classifier can degrade rapidly after it is deployed, as the adversary learns to defeat it. Currently the only solution to this is repeated, manual, ad hoc reconstruction of the classifier. In this paper we develop a formal framework and algorithms for this problem. We view classification as a game between the classifier and the adversary, and produce a classifier that is optimal given the adversary's optimal strategy. Experiments in a spam detection domain show that this approach can greatly outperform a classifier learned in the standard way, and (within the parameters of the problem) automatically adapt the classifier to the adversary's evolving manipulations.
Randomized Operating Point Selection in Adversarial Classification
Lecture Notes in Computer Science, 2014
Security systems for email spam filtering, network intrusion detection, steganalysis, and watermarking, frequently use classifiers to separate malicious behavior from legitimate. Typically, they use a fixed operating point minimizing the expected cost / error. This allows a rational attacker to deliver invisible attacks just below the detection threshold. We model this situation as a non-zero sum normal form game capturing attacker's expected payoffs for detected and undetected attacks, and detector's costs for false positives and false negatives computed based on the Receiver Operating Characteristic (ROC) curve of the classifier. The analysis of Nash and Stackelberg equilibria reveals that using a randomized strategy over multiple operating points forces the rational attacker to design less efficient attacks and substantially lowers the expected cost of the detector. We present the equilibrium strategies for sample ROC curves from network intrusion detection system and evaluate the corresponding benefits.
A dynamic-adversarial mining approach to the security of machine learning
Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery
Operating in a dynamic real world environment requires a forward thinking and adversarial aware design for classifiers, beyond fitting the model to the training data. In such scenarios, it is necessary to make classifiers-a) harder to evade, b) easier to detect changes in the data distribution over time, and c) be able to retrain and recover from model degradation. While most works in the security of machine learning has concentrated on the evasion resistance (a) problem, there is little work in the areas of reacting to attacks (b and c). Additionally, while streaming data research concentrates on the ability to react to changes to the data distribution, they often take an adversarial agnostic view of the security problem. This makes them vulnerable to adversarial activity, which is aimed towards evading the concept drift detection mechanism itself. In this paper, we analyze the security of machine learning, from a dynamic and adversarial aware perspective. The existing techniques of Restrictive one class classifier models, Complex learning models and Randomization based ensembles, are shown to be myopic as they approach security as a static task. These methodologies are ill suited for a dynamic environment, as they leak excessive information to an adversary, who can subsequently launch attacks which are indistinguishable from the benign data. Based on empirical vulnerability analysis against a sophisticated adversary, a novel feature importance hiding approach for classifier design, is proposed. The proposed design ensures that future attacks on classifiers can be detected and recovered from. The proposed work presents motivation, by serving as a blueprint, for future work in the area of Dynamic-Adversarial mining, which combines lessons learned from Streaming data mining, Adversarial learning and Cybersecurity.
Bayesian Games for Adversarial Regression Problems
2013
We study regression problems in which an adversary can exercise some control over the data generation process. Learner and adversary have conflicting but not necessarily perfectly antagonistic objectives. We study the case in which the learner is not fully informed about the adversary's objective; instead, any knowledge of the learner about parameters of the adversary's goal may be reflected in a Bayesian prior. We model this problem as a Bayesian game, and characterize conditions under which a unique Bayesian equilibrium point exists. We experimentally compare the Bayesian equilibrium strategy to the Nash equilibrium strategy, the minimax strategy, and regular linear regression.
A game-theoretic approach to assess adversarial risks
Risk Analysis IX, 2014
In our complex world today almost all critical infrastructures are interdependent and thus vulnerable to many different external and internal risks. To protect them against the greatest risks, a well-functioning risk management process is required to develop appropriate safety and security strategies. There are many wellestablished risk analysis methods existing. They predominantly apply empirical models and statistical data to quantify the risks. Within the realms of natural or aleatory risks this approach is considered suitable and functional. However, it could be a fatal flaw to apply such conventional, history-orientated models in order to assess risks that arise from intelligent adversaries such as terrorists, criminals or competitors. Approaches of classic risk analysis generally describe adversaries' choices as random variables, thus excluding the adversaries' behaviour and ability to adapt to security strategies. One possibility for considering human behaviour when analysing risks is the recourse to game theory. Game theory is the paradigmatic framework for strategic decision-making when two or more rational decision-makers (intelligent adversaries) are involved in cooperative or conflictive decision situations. In our study we propose an approach for combining a classic risk analysis method with a game-theoretic approach. Using a defenderoffender game as a basis, we simulate, exemplary for a terrorist attack against public transport, the behaviour and reactions (to applied security strategies of the defender) of a rational player acting as an adversary. Although risk analysis and game theory are very different methodologies, we show that linking them can significantly improve the quality of forecasts and risk assessments. If the behaviour and reactions of intelligent adversaries need to be considered, our approach contributes to enhance security through improving the allocation of scarce financial resources.
ACM SIGecom Exchanges, 2011
We consider the following setting: a decision maker should classify a finite set of data points with binary labels, minimizing the expected error. Subsets of data points are controlled by different selfish agents, which might misreport the labels in order to sway the decision in their favor. We design mechanisms (both deterministic and randomized) that reach an approximately optimal decision and are Strategy-Proof, i.e. agents are best off when they tell the truth. We examine the best approximation ratio that can be achieved using a Strategy-Proof mechanism in various conditions, thereby matching our upper bounds with lower ones. We show that when the approximation ratio is constant, our results can be casted into a classical machine learning classification framework, where the decision maker must learn an approximately optimal classifier based only on a sampled subset of the agents' points.
Security Theater: On the Vulnerability of Classifiers to Exploratory Attacks
2018
The increasing scale and sophistication of cyberattacks has led to the adoption of machine learning based classification techniques, at the core of cybersecurity systems. These techniques promise scale and accuracy, which traditional rule or signature based methods cannot. However, classifiers operating in adversarial domains are vulnerable to evasion attacks by an adversary, who is capable of learning the behavior of the system by employing intelligently crafted probes. Classification accuracy in such domains provides a false sense of security, as detection can easily be evaded by carefully perturbing the input samples. In this paper, a generic data driven framework is presented, to analyze the vulnerability of classification systems to black box probing based attacks. The framework uses an exploration exploitation based strategy, to understand an adversary's point of view of the attack defense cycle. The adversary assumes a black box model of the defender's classifier and ...
Detection games with a fully active attacker
2015 IEEE International Workshop on Information Forensics and Security (WIFS), 2015
We analyze a binary hypothesis testing problem in which a defender has to decide whether or not a test sequence has been drawn from a given source P 0 whereas, an attacker strives to impede the correct detection. In contrast to previous works, the adversarial setup addressed in this paper considers a fully active attacker, i.e. the attacker is active under both hypotheses. Specifically, the goal of the attacker is to distort the given sequence, no matter whether it has emerged from P 0 or not, to confuse the defender and induce a wrong decision. We formulate the defender-attacker interaction as a game and study two versions of the game, corresponding to two different setups: a Neyman-Pearson setup and a Bayesian one. By focusing on asymptotic versions of the games, we show that there exists an attacking strategy that is both dominant (i.e., optimal no matter what the defense strategy is) and universal (i.e., independent of the underlying sources) and we derive equilibrium strategies for both parties.
Detection in Adversarial Environments
Ieee Transactions on Automatic Control, 2014
We propose new game theoretic approaches to estimate a binary random variable based on sensor measurements that may have been corrupted by a cyber-attacker. The estimation problem is formulated as a zero-sum partial information game in which a detector attempts to minimize the probability of an estimation error and an attacker attempts to maximize this probability. While this problem can be solved exactly by reducing it to the computation of the value of a matrix, this approach is computationally feasible only for a small number of sensors. The two key results of this paper provide complementary computationally efficient solutions to the construction of the optimal detector. The first result provides an explicit formula for the optimal detector but it is only valid when the number of sensors is roughly smaller than two over the probability of sensor errors. In contrast, the detector provided by the second result is valid for an arbitrary number of sensor. While it may result in a probability of estimation error that is above the minimum achievable, we show that this error is small when the number of sensors is large, which is precisely the case for which the first result does not apply.