Anomaly Detection: Theory, Algorithms and APT War Game

Anomaly is a deviation from a normal behavior. Anomaly detection techniques are used to detect unusual patterns in data. These patterns deviate from the spectrum of normal behaviors in the data, and typically they represent critical events that occurred in the monitored system. For example, in Cyber security, anomaly detection can be used to identify sophisticated and targeted attacks like Advanced Persistent Threats (APT), where standard security systems often fail to detect.

The anomaly detection problem, in its most general form, is not easy to solve. In fact, most of the existing anomaly detection techniques solve a specific formulation (instance) of the problem. The formulation is induced by various factors such as the nature of the data, availability of labeled data, type of anomalies to be detected, etc. Often, the application domain in which the anomalies have to be detected determines these factors.

Usually, in addition to the challenge of detecting anomalies in a dataset, the analyzed data is also high dimensional, which makes it more difficult to analyze and interpret. For example, email traffic can be represented by thousands of textual and numeric features.

Businesses in all sectors (military, intelligence, governmental, industrial etc.) can benefit from anomaly detection. Data collected and stored in databases and warehouses is data that represents some real world processes. Anomalies and outliers, which exist in the real world processes, will be captured with the collected data. The application of the appropriate technique to identify and detect these anomalies can lead to new knowledge about the data and hence the real world process.

Anomaly detection is applicable in a variety of domains, such as intrusion detection, fraud detection, fault detection, system health monitoring, event detection in sensor networks, and detecting eco-system disturbances.

In this course, we will focus on theoretical and practical anomaly detection for network and host security, intrusion detection and identification of malicious activities.

Course Structure
In the first part of this course (~1.5 days), we will introduce the concept of anomalies, provide motivation for anomaly detection and explore several real-world use cases of anomalies. We will overview different data types, including high-dimensional data, and provide methods for pre-processing of data. We will explore the different categories of anomaly detection and the different types of anomalies. We will conclude this section with methods for evaluation of anomaly detection methods.

In the second part of this course (~2.5 days), we will survey different techniques for anomaly detection. For example, classification based techniques, nearest neighbors based techniques, clustering based techniques and spectral based techniques. For each technique, we will explore the theory behind it, its different categories, the pros and cons, demos and practice.

In the third part of the course (1 day), we will conclude the course with a cyber security war game, where the blue teams will act as the defense teams and the red team will act as the attacking team. The defense teams will construct an anomaly based intrusion detection system and the attacking team will try to bypass the defense system. In this part, we will practice in real-life different methods for anomaly detection.

5 days


Part I: Anomalies, data, categories, types and evaluation
Data types
Preprocessing of data
Anomaly detection categories
Anomalies types
Anomaly detection result
Evaluation of anomaly detection

Part II: Anomaly detection techniques
Classification based techniques
Nearest neighbors based techniques
Clustering based techniques
Spectral based techniques

Part III: Cyber war game
APT - Advanced Persistent Threat
Cyber security scenario
Operations order
Blue team - construct the defense system
Red team - build the attacks
War game - red team vs. blue team
© Brainstorm Private Consulting. All Rights Reserved
Designed By : Template World