Evaluation Metrics for Predictive Monitoring Systems with Highly Imbalanced Datasets

Abstract

A predictive monitoring system is a machine learning model used periodically with the goal of monitoring the behaviour of database entities. Most monitoring systems are trained and tested on highly imbalanced data as the target events are quite rare. Moreover, the evaluation of predictive monitoring is even more complicated by aspects specific to the task (e.g. proper timing of alerts and possible reoccurrence of alerts). Thus, there is a need in stable, class imbalance tolerant metrics that also reflect all monitoring-specific issues. We have investigated existing approaches of monitoring systems evaluation and found them to be quite case-specific. Therefore, we have extended and modified the methods in use to be domain-independent and easily adjustable to the task at hand. The proposed evaluation approach is implemented and evaluated with experiments on data from different domains. In addition, we analysed several metrics designed specifically for imbalanced data to conclude if they can be used for monitoring evaluation due to restrictions of the approach.