Preoptimistične ocene točnosti napovednih modelov - Tupkušić, Mirza (Informatica Medica Slovenica, 2022)
Prediction models use various statistical methods for building classification rules to classify units into pre-specified groups based on the learning data. In practice, the data are often not suitable for the chosen procedure and they need to be pre-processed before training the classifier. An important example are imbalanced data where the naive approach can lead to poor accuracy for the minority class. Many data augmentation approaches have been developed to alleviate this issue. However, when using these techniques, one needs to be careful to correctly evaluate the performance of the classifier in terms of its predictive accuracy, because incorrect evaluation can lead to an overly optimistic estimate of the classifier’s performance. We explain in detail why this happens and showcase the different contributing factors. The results are illustrated using various performance measures, various data augmentation techniques, and various cross-validation techniques. Our results can help the developers of prediction models to correctly evaluate predictive ability of the derived model, as well as to understand and critically appraise whether the predictive ability of the model was correctly estimated or the evaluation was too optimistic.
Keywords: cross-validation; napovedni model; navzkrižno preverjanje; neuravnoteženi podatki; prediction models; rare events