Anomaly Detection Information
Anomaly detection, also referred to as outlier detection[1] refers to detecting patterns in a given data set that do not conform to an established normal behavior.[2] The patterns thus detected are called anomalies and often translate to critical and actionable information in several application domains. Anomalies are also referred to as outliers, surprise, aberrant, deviation, peculiarity, etc.
Three broad categories of anomaly detection techniques exist. Supervised anomaly detection techniques learn a classifier using labeled instances belonging to normal and anomaly class, and then assign a normal or anomalous label to a test instance. Semi-supervised anomaly detection techniques construct a model representing normal behavior from a given normal training data set, and then test the likelihood of a test instance to be generated by the learnt model. Unsupervised anomaly detection techniques detect anomalies in an unlabeled test data set under the assumption that majority of the instances in the data set are normal.
Contents |
Applications
Anomaly detection is applicable in a variety of domains, such as intrusion detection, fraud detection, fault detection, system health monitoring, event detection in sensor networks, and detecting eco-system disturbances. It is often used in preprocessing to remove anomalous data from the dataset.
Popular Anomaly Detection Techniques
Several anomaly detection techniques have been proposed in literature. Some of the popular techniques are:
- Distance based techniques (k-nearest neighbor, Local Outlier Factor).
- One Class Support Vector Machines.
- Replicator Neural Networks.
- Cluster analysis based outlier detection.
Application to Data Security
Anomaly detection was proposed for Intrusion detection systems (IDS) by Dorothy Denning in 1986.[3] Anomaly detection for IDS is normally accomplished with thresholds and statistics, but can also be done with Soft computing, and inductive learning[4]. Types of statistics proposed by 1999 included profiles of users, workstations, networks, remote hosts, groups of users, and programs based on frequencies, means, variances, covariances, and standard deviations.[5] The counterpart of Anomaly detection in Intrusion detection is Misuse Detection.
See also
References
- ^ Hans-Peter Kriegel, Peer Kröger, Arthur Zimek (2009). "Outlier Detection Techniques (Tutorial)". 13th Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD 2009) (Bangkok, Thailand). http://www.dbs.ifi.lmu.de/Publikationen/Papers/tutorial_slides.pdf. Retrieved 2010-06-05.
- ^ Varun Chandola, Arindam Banerjee, and Vipin Kumar, Anomaly Detection: A Survey, ACM Computing Surveys, Vol. 41(3), Article 15, July 2009
- ^ Denning, Dorothy, "An Intrusion Detection Model," Proceedings of the Seventh IEEE Symposium on Security and Privacy, May 1986, pages 119-131.
- ^ Teng, Henry S., Chen, Kaihu, and Lu, Stephen C-Y, "Adaptive Real-time Anomaly Detection Using Inductively Generated Sequential Patterns," 1990 IEEE Symposium on Security and Privacy
- ^ Jones, Anita K., and Sielken, Robert S., "Computer System Intrusion Detection: A Survey," Technical Report, Department of Computer Science, University of Virginia, Charlottesville, VA, 1999
Categories: Data mining | Data security | Statistical outliers
|