690 KB0 KB200220242022Blagus, RokTupkušić, Mirzaštevilka:1/2letnik:27str. 1-13ISSN:1318-2129COBISSID_HOST:152443651URN:URN:NBN:SI:doc-S1M4KA4DslSlovensko društvo za medicinsko informatikoInformatica Medica Slovenicacross-validationnapovedni modelnavzkrižno preverjanjeneuravnoteženi podatkiprediction modelsrare eventsPreoptimistične ocene točnosti napovednih modelov| an illustration based on the joint use of sampling techniques and cross-validation| ilustracija na primeru skupne uporabe tehnik vzorčenja in navzkrižnega preverjanja| Over-optimistic assessment of the performance of prediction models|Prediction models use various statistical methods for building classification rules to classify units into pre-specified groups based on the learning data. In practice, the data are often not suitable for the chosen procedure and they need to be pre-processed before training the classifier. An important example are imbalanced data where the naive approach can lead to poor accuracy for the minority class. Many data augmentation approaches have been developed to alleviate this issue. However, when using these techniques, one needs to be careful to correctly evaluate the performance of the classifier in terms of its predictive accuracy, because incorrect evaluation can lead to an overly optimistic estimate of the classifier’s performance. We explain in detail why this happens and showcase the different contributing factors. The results are illustrated using various performance measures, various data augmentation techniques, and various cross-validation techniques. Our results can help the developers of prediction models to correctly evaluate predictive ability of the derived model, as well as to understand and critically appraise whether the predictive ability of the model was correctly estimated or the evaluation was too optimisticTEXTznanstveno časopisjejournalsSlovenian National E-content AggregatorNational and University Library of SloveniaSlovensko društvo za medicinsko informatiko