On understanding normal protocol behaviour to monitor and mitigate the abnormal (original) (raw)

Towards model-based anomaly detection in network communication protocols

2016 2nd International Conference on Frontiers of Signal Processing (ICFSP), 2016

Over the last few years many techniques have been applied to find and mitigate vulnerabilities, misuses, cyberattacks and other cyber-security flaws. One of the approaches, which we consider in this paper, is a model-based technique applied to network communication protocols. This idea is not brand new, and model-based techniques have been successfully used to verify and validate the standard models of communication protocols. However, the implementation of network protocols varies from one system to another, and in many cases they miss standards or recommendations. Attackers know these flaws very often and try to use them before everybody else finds them, what can be called "zero-day exploit of communication protocol." To address this issue, a combination of the best features of model-based and anomaly detection techniques could be applied. Treating discovered anomalies as a signature of a cyber-attack or any other malicious activity and focusing on the investigation of them could significantly increase the success rate of the defense against them. In this paper we considered some significant inputs from the research community to model-based anomaly detection in network communication protocols. Then we prepared a synthetic brief of the theories and methods for modelling network protocols as state-machines. Next we examined the application of it in a cyber-security area. Finally we proposed some key directions that actual research should follow to bring some breakthrough results as soon as possible.

Attacks Detection Based on IP and TCP Protocols Violation

The International Journal of Forensic Computer Science, 2006

One of the biggest challenges in the network intrusion detection field is the limitation imposed by the use of well-known attack signatures that disable the previous detection of new attacks. This work presents a packet analysis methodology for detecting anomalous behaviors, not based on attack signatures, but on verifying whether the network protocols are being violated, and on the content of the respective headers. The biggest benefit of this methodology is the possibility of detecting anomalies or inadequate behaviors that can correspond, totally or partially, to variations on well-known and unknown attacks.

Crosstalk: A Scalable Cross-Protocol Monitoring System for Anomaly Detection

2010

Monitoring is crucial both to the correct operation of a network and to the services that run on it. Operators perform monitoring for various purposes, including traffic engineering, quality of service, security and detection of faults and mis-configurations. However, the relentless growth of IP traffic volume renders real-time monitoring and analysis of data a very challenging problem. In this paper we introduce Crosstalk, a scalable and efficient distributed monitoring architecture that uses cross-protocol correlation to detect network anomalies. While applicable to a wide range of applications such as botnet detection, spam mitigation and mis-configurations, we pick a point in this application space, concentrating on VoIP attacks. We present extensive simulation results based both on generated calls and on millions of Call Data Records (CDRs) from a large VoIP operator to show our approach's performance and effectiveness.

Attacks & Defense Mechanisms for TCP/ IP Based Protocols

TCP/IP protocol suite is the most widely used communication protocol and has become the de facto standard for internet based communications. It is a set of robust protocols originally designed to provide reliable communication services that allow co-operating computers to share resources across networks.

A Network Traffic Representation Model for Detecting Application Layer Attacks

International Journal of Computing and Digital Systems, 2016

Intrusion Detection Systems (IDS) play an important role in network security, protecting systems and infrastructures from malicious attacks. With the emerging of novel threats and offensive mechanisms, IDS require updates in order to efficiently detect new menaces. In this paper we propose an anomaly-based detection model designed for particular application protocols, exploited by emerging menaces known as Slow Denial of Service (DoS) Attacks. We define parameters characterizing network traffic and we describe in detail how to extrapolate them from a network traffic capture. We motivate the need of packet inspection in certain contexts in order to retrieve correct data. We analyze and describe how the proposed model behaves on two real scenarios involving legitimate and malicious activities, respectively. Thanks to our model, a detection framework for attacks working at the application layer of the communication protocol stack is provided, allowing and facilitating the execution of detection algorithms. Indeed, though the adoption of such framework, the design of efficient detection systems is simplified and designers work is reduced, allowing them a faster deploy of efficient detection algorithms. The aim of this paper is to provide an effective framework for application DoS attacks detection.

Retaliation against protocol attacks

Journal of Information Assurance and Security, 2008

Abstract: Security protocols intend to give their parties reasonable assurance that certain security properties will protect their communication session. However, the literature confirms that the protocols may suffer subtle and hidden attacks. Flawed protocols are customarily sent back to the design process, but the costs of reengineering a deployed protocol may be prohibitive. This paper outlines the concept of retaliation: who would steal a sum of money today, should this pose significant risks of having twice as much stolen back tomorrow? ...

Verify Results of Network Intrusion Alerts Using Lightweight Protocol Analysis

21st Annual Computer Security Applications Conference (ACSAC'05)

We propose a method to verify the result of attacks detected by signature-based network intrusion detection systems using lightweight protocol analysis. The observation is that network protocols often have short meaningful status codes saved at the beginning of server responses upon client requests. A successful intrusion that alters the behavior of a network application server often results in an unexpected server response, which does not contain the valid protocol status code. This can be used to verify the result of the intrusion attempt. We then extend this method to verify the result of attacks that still generate valid protocol status code in the server responses. We evaluate this approach by augmenting Snort signatures and testing on real-world data. We show that some simple changes to Snort signatures can effectively verify the result of attacks against the application servers, thus significantly improve the quality of alerts.

Behavioral Model to Detect Anomalous Attacks in Packet Transmission

Inside a network environment, packets is the most important in carrying data to perform communication. Such a circumstance is easy to be attacked by an intruder and perform eavesdropping which leads to data loss/duplication/redundancy. Comprehend speaking, packet dropping and modification are the two common attacks that can be easily launched by an adversary to disrupt communication in multi hop networks, specifically mobile ad hoc networks. Hence a remedial approach is proposed to compensate such attacks. A tree based approach is designed to designate the attack in order to identify packet droppers and modifiers. In this direction, it has been assumed that the mobile nodes continuously monitor the behaviors of the forwarding mobile nodes which may be neighbors to determine if their neighbors are misbehaving. To address this problem, a hierarchical method is proposed and detects malicious mobile nodes that drop or modify packets. Extensive analysis and simulations have been conducted to study the performance of attacks with respect to efficiency of the scheme.

Network Intrusion Detection: Evasion, Traffic Normalization, and End-to-End Protocol Semantics

A fundamental problem for network intrusion detection systems is the ability of a skilled attacker to evade detection by exploiting ambiguities in the traffic stream as seen by the monitor. We discuss the viability of addressing this problem by introducing a new network forwarding element called a traffic normalizer. The normalizer sits directly in the path of traffic into a site and patches up the packet stream to eliminate potential ambiguities before the traffic is seen by the monitor, removing evasion opportunities. We examine a number of tradeoffs in designing a normalizer, emphasizing the important question of the degree to which normalizations undermine end-to-end protocol semantics. We discuss the key practical issues of "cold start" and attacks on the normalizer, and develop a methodology for systematically examining the ambiguities present in a protocol based on walking the protocol's header. We then present norm, a publicly available user-level implementation of a normalizer that can normalize a TCP traffic stream at 100,000 pkts/sec in memory-to-memory copies, suggesting that a kernel implementation using PC hardware could keep pace with a bidirectional 100 Mbps link with sufficient headroom to weather a high-speed flooding attack of small packets. © extent of normalization vs. protection © impact on end-to-end semantics (service models) © impact on end-to-end performance © amount of state held ©

A syntactic approach for identifying multi-protocol attacks

2009 International Conference on Ultra Modern Telecommunications & Workshops, 2009

In the context of multiple security protocols running in the same environment, we propose a syntactical approach for identifying multi-protocol attacks. The proposed approach uses a canonical security protocol model, where terms that can be verified by protocol participants are denoted by canonical terms. In order to enable the identification of subtle "type-flaw" attacks, where terms can be substituted with other types of terms, we introduce a canonical identifier. The approach is validated by analyzing several security protocol pairs. The attacks discovered by our approach are also discovered by existing security protocol verification tools.