Glitches, Blips, and Outliers. Real-Time Anomaly Detection Using Machine Learning

12th June 2017

Written by Rebecca Herson

Part 2

It’s About Time.

As mentioned in our previous post, a bad line of code in a software update may be causing your company to lose money every second due to cart abandonment on a website. Identifying the problem is the first step in resolving it, but only if you know where to begin looking for the problem in the first place. And, the time that elapses between when the problem first occurs and when the problem is detected can mean the difference between a few hundred dollars and millions.

Realistically, traditional anomaly detection solutions—manual research, dashboards, and alerting systems—take time and require human analysis and processing. If you are managing just a couple of business metrics or KPIs, then solving a problem quickly could be feasible. Unfortunately, this manual approach, is not scalable to thousands or millions of metrics while maintaining a real-time response.

Aren’t All Anomaly Detection Systems Alike?

There are two main methods of machine learning: supervised and unsupervised. With supervised machine learning, humans feed datasets containing pre-labeled and categorized examples into the algorithm to develop a general model for each category. Then, the algorithm processes the real un-categorized data and attempts to put each item into one of the pre-learned categories.

A supervised machine learning algorithm cannot place an item into a category it doesn’t know, since the algorithm can only interpret information based on the human-provided data. This type of system is unrealistic for large-scale anomaly detection, since the algorithm would need examples of every single anomaly, data distribution, pattern, and trend possibility.

In contrast, unsupervised machine-learning algorithms learn the basis for “normal,”, and then apply a statistical test to determine if a specific data point is an anomaly. This type of anomaly detection system is able to detect any type of anomaly, including new ones. However, challenges arise when it becomes necessary to decide what constitutes “normal” for the time series being analyzed.

Anodot uses a hybrid “semi-supervised” real-time machine learning system. While the vast majority of the classifications are completed using unsupervised methods, customers are encouraged to provide feedback, which can then be incorporated as additional variables and data points in the system.

Adaptation is Key

Anodot’s algorithms are adaptive, which means they adjust to and accept time series changes when the range of normal alters. Going back to our dog walker metaphor, this is analogous to you not seeing your neighbor walking his dog each morning for several weeks in a row. Eventually, you adapt your mental model about your neighbor: he doesn’t walk his dog at 7 AM every morning. This perception and interpretation becomes the new normal.

Anodot’s adaptive learning algorithms does the same way by assigning to anomalies an increasing ability to change the “normal” model the longer that anomaly persists. The result is a real-time automated anomaly detection method which flags anomalies and adapts to pattern changes.

Intelligent Real-time Autonomous Machine Learning

When anomalies are discovered, companies are presented with business insight that can help them discover relationships between metrics and variables so data can be distilled down to a manageable number of correlated incidents, which can then be investigated by human experts.

By filtering out the massive amount of insurmountable data and pinpointing the issues at hand, Anodot can offer businesses meaningful insights, empowering organizations to turn issues into opportunities.

ABOUT ANODOT

Anodot was founded in 2014, and since its launch in January 2016 has been providing valuable business insights through anomaly detection to its customers in fintech, ad-tech, web apps, mobile apps, e-commerce and other data-heavy industries. Over 40% of the company’s customers are publicly traded companies, including Microsoft, Waze (a Google company), AppNexus, and many others. Anodot’s real time business incident detection uses patented machine learning algorithms to isolate and correlate issues across multiple parameters in real time, supporting rapid business decisions. Learn more at:  http://www.anodot.com/.

 


Tags:
Blogs

BACK TO BLOGS