Evaluation Metrics for Predictive Monitoring Systems with Highly Imbalanced Datasets
Kuupäev
2020
Autorid
Ajakirja pealkiri
Ajakirja ISSN
Köite pealkiri
Kirjastaja
Tartu Ülikool
Abstrakt
A predictive monitoring system is a machine learning model used periodically
with the goal of monitoring the behaviour of database entities. Most monitoring
systems are trained and tested on highly imbalanced data as the target events are quite
rare. Moreover, the evaluation of predictive monitoring is even more complicated by
aspects specific to the task (e.g. proper timing of alerts and possible reoccurrence of
alerts). Thus, there is a need in stable, class imbalance tolerant metrics that also reflect
all monitoring-specific issues.
We have investigated existing approaches of monitoring systems evaluation and found
them to be quite case-specific. Therefore, we have extended and modified the methods
in use to be domain-independent and easily adjustable to the task at hand. The
proposed evaluation approach is implemented and evaluated with experiments on data
from different domains. In addition, we analysed several metrics designed specifically
for imbalanced data to conclude if they can be used for monitoring evaluation due to
restrictions of the approach.
Kirjeldus
Märksõnad
Predictive monitoring, machine learning, evaluation metrics, class imbalance